How AI Search Crawling Actually Works: Truths About AI

Craig Kanalley
Tue, 28 Oct 2025 18:40:03 +0000
Ai, Technical seo

Don’t be fooled by the magic tricks of the AI era. There are companies selling things that they can’t deliver, disappearing acts of hallucinations and illusions, and worst of all, the big tech companies are accelerating the race to the bottom.

Those were some of the messages from Jamie Indigo of Cox Automotive in a Moz webinar entitled, “The Dark Side of AI.” She shared words of wisdom and cautionary tales, as Chima Mmeje of Moz SEO hosted the lively discussion. In a nutshell: The digital industry needs to put the brakes on how we’re thinking and talking about AI.

One of the biggest themes that Indigo emphasized during her talk was the fact that AI search is not the same as traditional search engines. While traditional search is about information retrieval, AI systems are language prediction models. That’s why they’re prone to hallucinations.

AI systems are also designed to use as few tokens as possible in their responses, which saves their companies money, and those shortcuts can provide disappointing or faulty responses to users (even if the users don’t realize it).

Keep in mind, as Indigo pointed out:

AI doesn’t have a true search index.

AI search results aren’t rankings. They’re volatile, in some cases shifting multiple times in a single day!

AI isn’t a search engine, even though it draws from them. Some of the same sources that power OpenAI also feed Google Gemini.

Webinar by Moz: The Practical Marketer Series
Guest: Jamie Indigo, Cox Automotive (LinkedIn)
Host: Chima Mmeje, Moz (LinkedIn | Twitter/X)

?A Power Play

Indigo says that what’s really happening is these big tech companies in the AI space are positioning themselves for a massive power grab. They claim to be larger and more powerful and transformative than they truly are.

The AI hype creates FOMO (fear of missing out), which leads to a cycle or loop of more hype. This quickly results in POMO (profiting on missing out). Investors are pouring literally millions of dollars into AI projects that have yet to show real profit.

That leads to a black box optimization era, where marketers are “feeding” systems that they don’t really understand. The opacity of these systems makes it easy for companies to overpromise and underdeliver.

?How AI Crawling Works

Let’s take a step back. AI bots don’t crawl the web like Google does. In fact, the process varies dramatically depending on the model.

Here’s what you need to know, according to Indigo:

ChatGPT can’t crawl JavaScript.

Try turning off JavaScript in Chrome to see what ChatGPT might be missing on your site. You can also run a Screaming Frog crawl without JavaScript for comparison.

Google’s LLMs have no such limitation.

However, the LLM crawls are currently kept separate from Google’s main search to prevent “AI slop” from contaminating traditional search results.

What ChatGPT does crawl:

Domain credibility

Freshness (how recently something was published)

Meta descriptions and titles

ChatGPT’s own support team has confirmed the aforementioned as primary signals. Other AI systems, like Perplexity, have their own crawling logic — but the fundamentals still matter: titles, meta tags, clean HTML, and crawlable links.

In short, AI bots aren’t magic. They still depend on solid technical SEO foundations, as our CEO and founder at RebelMouse Andrea Berry has emphasized as well.

?So Now What?

As we navigate this evolving space, we need to proceed with caution. Already, there are legal and ethical warning signs.

So why do SEOs remain obsessed with optimization? First, Indigo points out it’s in our job title.

But it’s also in our nature to make things work better. When it comes to AI, the goal isn’t always “accuracy” — it’s believability.

Here’s how to stay grounded:

1. Push back against the “magic” narrative. Recognize FOMO when you see it in sales pitches.

2. Use clear language. Don’t overhype technology you’re still learning about.

3. Stay calm and analytical. Replace panic with reasoned experimentation.

?The Hidden Risks of AI-Generated Content

If you’re using AI to create content at scale, you should definitely consider stopping.

Be especially careful with content creation. If your pages look and read like everything else produced by LLMs, there’s no reason for crawlers, or users, to engage.

Low-quality, derivative content can be excluded from crawling altogether. Consider implementing an LLMs.txt file to block unwanted AI crawlers and protect your data.

?Jamie’s Perspective: The Problem With AI

“AI was built on our intellectual property.” —Jamie Indigo

Plain and simple. That sums it up. The technology isn’t as groundbreaking as the marketing suggests. The “wow” factor often hides the fact that much of AI’s foundation comes from existing human-created work. Beneath the buzzwords, many of these conversations circle back to the basics of technical SEO, such as structure, clarity, and human-centered design.

?Final Advice: Build for Humans, Not Machines

For smaller sites especially, Indigo offers a simple truth:

“Be unique. Be human-centric. When users get burned out on AI everything, they’ll remember your name. That’s how you last.”

In the end, AI might just consume itself. A snake that eats itself is the analogy Jamie used. As you may have heard, the web is being filled up with recycled “AI slop” (AI content being trained on AI content, generating new AI content on top of it — and facts can get lost in this game of telephone). But genuine creativity, expertise, and human touch will always stand apart.