Not all AI bots are created equal: what those crawlers in your logs actually mean

Open up your server logs, or the bot report in whatever analytics tool you use, and you will find AI companies crawling your website. Most site owners have one of two reactions. Either a small thrill, because ChatGPT knows we exist. Or a flash of irritation, because nobody asked permission.

Both reactions treat “AI bots” as one thing. It isn’t one thing. It’s at least four, and the difference decides whether that line in your log is worth anything to your business.

Everything below comes from the companies’ own published documentation, which we read on 14 September 2026. That date matters more than it usually would. One of these bots was renamed a few weeks ago and half the guides online haven’t caught up.

Here is the example that makes it click. OpenAI doesn’t run one bot. It runs four, and three of them turn up in ordinary website logs. GPTBot is a training crawler. If you see it, OpenAI has been reading your content to help train its models. It will never send you a visitor. That isn’t a criticism, it’s the design: a training crawler collects text, it doesn’t refer people.

ChatGPT-User is the opposite. Somebody asked ChatGPT a question a few seconds ago, and ChatGPT came to your page to answer it. There is a human at the other end of that request, right now, waiting. That is the hit that turns into a citation and sometimes a click.

Same company. Two bots. Two completely different meanings. And, importantly, two separate decisions you can make about them.

OpenAI says this out loud in its own documentation:

“Each setting is independent of the others - for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models.”

OpenAI, Overview of OpenAI Crawlers, checked 14 September 2026

That sentence is the whole article in miniature. The bots are separable. Most advice you’ll read online treats them as a single blob to be allowed or blocked, and that advice will cost you.

The four jobs an AI bot can be doing

Before the per-platform detail, here is the mental model. Every AI bot that hits your site is doing one of four jobs.

Training. Collecting your content into the pile a model learns from. No traffic to you, now or later, at least not directly. The benefit, if there is one, is diffuse and deferred.

Live user-triggered fetch. A person just asked a question and the assistant is fetching your page this second to answer it. This is the one that correlates with being cited, and with referral clicks.

AI-search index. Building the searchable index that an AI answer engine queries when it needs sources. The crawl itself isn’t a visitor, but it’s the precondition for ever being quoted. Block this and you are not in the answers.

Preview, unfurl and operations. Link preview cards when somebody shares your URL in a chat, ad landing-page safety checks, that sort of thing. No traffic, and none intended.

Keep those four in your head and the rest of this makes sense. Here’s the short version:

Now, platform by platform.

OpenAI: four bots, three meanings

OpenAI is the clearest documented of the lot, which makes it a good place to start.

GPTBot is training. OpenAI’s description: content “that may be used in training our generative AI foundation models”. Seeing it means your content has been ingested. It means nothing about traffic.

One quirk worth knowing. When GPTBot fetches your robots.txt file, OpenAI may add a robots.txt marker into the user-agent string. So a GPTBot line with an extra robots.txt; in it is the same bot checking your rules, not a second mystery bot.

OAI-SearchBot is the AI-search index, “used to surface websites in search results in ChatGPT’s search features”. This is the one most people block by accident and really shouldn’t. OpenAI is blunt about the consequence: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

If you want to be visible in ChatGPT, this is the bot that gets you there.

ChatGPT-User is the live fetch, and it’s your traffic signal. OpenAI: “When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent.” It is explicitly not an automatic crawler. A human was in the loop.

If you want one number to watch from all of this, watch this one. Our read: a rising ChatGPT-User count is the closest thing in the whole table to a real audience signal, even though it won’t show up as a session in your analytics.

Two things OpenAI states that trip people up. First, blocking ChatGPT-User does not remove you from ChatGPT search, and allowing it does not put you in. That’s OAI-SearchBot’s job. Second, and this matters: “Because these actions are initiated by a user, robots.txt rules may not apply.” That’s a vendor telling you in writing that your robots.txt line here is advisory. Anyone who thinks they have firmly blocked ChatGPT by adding a ChatGPT-User disallow has not necessarily done so.

OAI-AdsBot is the fourth one, and it’s missing from nearly every article on this topic. It checks the safety of pages submitted as ads on ChatGPT, and OpenAI confirms its data is not used for model training.

Here’s the useful bit. It only visits pages that were submitted as ads. So if OAI-AdsBot shows up in your logs and you are not running ChatGPT ads, somebody has submitted one of your URLs as an ad landing page. Worth a look.

Also handy: if you change your robots.txt, OpenAI says “it can take ~24 hours from a site’s robots.txt update for our systems to adjust”. Don’t conclude your change failed on day one.

Google: the one with no user agent

Google is messier, because Google has been crawling the web for twenty-odd years and the AI bots were bolted onto an existing fleet.

Google sorts its own crawlers into three groups, and the sorting criterion is exactly the distinction this article is built on: who asked for the fetch, and whether robots.txt is binding. Note the wording gradient, which is unusually honest for a vendor.

• Common crawlers: “They always respect robots.txt rules for automatic crawls”
• Special-case crawlers: “may or may not respect robots.txt rules”
• User-triggered fetchers: “Because the fetch was requested by a user, these fetchers ignore robots.txt rules”

Always respect, may or may not, ignore. That’s the spectrum.

Google-Extended is the most misunderstood entry on this whole page, and it is not a bot. It has no user agent at all. Google’s words: “Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.”

So you will never see Google-Extended in your logs. If you think you have, it’s fake. The actual fetching is done by Googlebot. Google-Extended is a switch you flip in robots.txt, nothing more.

And the trap: it isn’t a training-only switch. It controls whether your content is used for training Gemini and for grounding, which Google defines as “providing content from the Google Search index to the model at prompt time to improve factuality and relevancy”. Grounding is the mechanism by which Gemini pulls your page in at answer time and cites you.

Turn off Google-Extended because you dislike the training, and you have also removed yourself from Gemini’s grounded answers. A lot of people have paid that price without realising they were buying anything. The one consolation: Google confirms Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”. Classic search is untouched. The entire cost lands on the AI side.

Googlebot is the familiar one, still doing search indexing, and also the agent that actually collects content for Gemini training and grounding unless Google-Extended tells it otherwise.

One practical warning if you’re building a log filter. Google’s published user-agent strings contain Chrome/W.X.Y.Z as a literal placeholder, and Google says so: “If you are searching your logs or filtering your server for a user agent with this pattern, use wildcards for the version number rather than specifying an exact version number.” Filter on the exact string and you will match nothing at all, then conclude Googlebot never visits. It does.

Google-GeminiNotebook is a live user-triggered fetch, and this one deserves flagging because it was recently renamed. It used to be Google-NotebookLM. Google now lists that as the “Former agent (supported until August 2026)”.

Every guide written before mid-2026, and every AI answer drawing on older material, will tell you the agent is Google-NotebookLM. Increasingly, it isn’t. If you set up a log filter on the old name, it will quietly stop matching.

What it does: “requests individual URLs that Gemini Notebook users have provided as sources for their projects”. No click arrives, but somebody deliberately typed your URL into their research notebook. Our take: that’s a stronger signal your content matters than any training crawl, even though it produces precisely zero sessions in GA4.

Google-Agent is the interesting one nobody’s writing about. Google describes it as “used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request”.

Read that again: perform actions. An AI agent transacting on your site on somebody’s behalf. If you run e-commerce or take bookings, that isn’t a crawl, it’s potentially a customer with no human on the other end of your analytics. This is our assessment rather than anything Google has spelled out, but the direction of travel seems clear enough.

Google-CloudVertexBot is narrow: it handles crawls “requested by the site owners” for building Vertex AI Agents, and has “no effect on Google Search or other products”. Same logic as OAI-AdsBot. It’s meant to be something you asked for. Seeing it unexpectedly is worth a question.

Google-Read-Aloud fetches pages and reads them out using text-to-speech. No session in your analytics, but a real person is consuming your content by ear. Blocking it is an accessibility decision, not an AI decision. Anyone blanket-blocking anything beginning with “Google-” should know that. (It also has a former name, google-speakr, now deprecated.)

GoogleOther is the generic bucket, “used by various product teams for fetching publicly accessible content”, for example “one-off crawls for internal research and development”. Google says GoogleOther crawl preferences “don’t affect any specific product”, which is deliberately vague. We are not going to tell you whether it feeds AI training, because Google doesn’t say and we didn’t measure it. There are also GoogleOther-Image/1.0 and GoogleOther-Video/1.0 variants.

A few smaller Google fetchers you may spot, all user-triggered: GoogleMessages (link previews for URLs shared in chats), FeedFetcher-Google (RSS and Atom), Google-CWS (Chrome Web Store), Google-Pinpoint, GoogleProducer and Google-Site-Verification.

Worth quoting Google’s own caveat on that list: “The list is not exhaustive, it only covers the requestors that are more likely to show up in log files and that we’ve received questions about.” Followed by: “Caution: The user agent string can be spoofed.”

One last Google note. All of this documentation has moved, from developers.google.com/search/docs/crawling-indexing/... to developers.google.com/crawling/docs/crawlers-fetchers/.... When we checked, the old path for the user-triggered fetchers page returned a 404 and only the new path worked. If you’re reading an article that links to the old location, you’re reading an article nobody has checked in a while.

Anthropic: same three-way split, one important difference

Anthropic runs the same three-way structure as OpenAI, and describes the reasoning in almost identical language: “Anthropic uses different robots to enable website owner transparency and choice.”

Two vendors independently landing on the same taxonomy is decent evidence that the split is real, rather than one company’s documentation quirk.

ClaudeBot is training, “collecting web content that could potentially contribute to their training”. No traffic.

One honest detail about blocking it. Anthropic says a block “signals that the site’s future materials should be excluded from our AI model training datasets”. Note future. Blocking is forward-looking. It does not retract what’s already been collected, and Anthropic deserves some credit for saying so plainly rather than implying otherwise.

Claude-User is the live fetch: “When individuals ask questions to Claude, it may access websites using a Claude-User agent.” This is Anthropic’s traffic signal, the direct counterpart to ChatGPT-User.

And here is a genuine difference between the two companies that we haven’t seen written up anywhere. OpenAI says robots.txt “may not apply” to its user-triggered fetches. Anthropic makes no such exception. It states flatly that “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt”, and describes Claude-User as letting site owners control “which sites can be accessed through these user-initiated requests”.

Same class of bot, opposite stated policy. If you block Claude-User expecting it to stick, Anthropic says it will. If you block ChatGPT-User expecting the same, OpenAI has told you it might not.

Anthropic also spells out the cost of blocking: it “prevents our system from retrieving your content in response to a user query, which may reduce your site’s visibility for user-directed web search”.

Claude-SearchBot is the index builder, which “navigates the web to improve search result quality for users”. Blocking it “prevents our system from indexing your content for search optimization, which may reduce your site’s visibility and accuracy in user search results”.

One transparency gap, in fairness to the reader: Anthropic publishes the bot names and robots.txt tokens but, unlike OpenAI and Google, does not publish full example user-agent strings. So we’re not printing any. If you see a full Claude user-agent string quoted confidently in somebody’s blog post, ask where they got it.

A useful Anthropic-specific bonus: it supports the non-standard Crawl-delay directive in robots.txt, with the example User-agent: ClaudeBot / Crawl-delay: 1. Google ignores Crawl-delay entirely, so this is one of the few places it does something.

One counterintuitive warning from Anthropic that applies to every vendor on this page. Do not block these bots at the IP or firewall level. Anthropic’s reasoning: “Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.”

Think that through. If you firewall the bot, it can’t fetch your robots.txt, so it can never learn that you wanted to opt out. A firewall block is not a stronger version of a robots.txt block. It’s a different thing, and in some ways a weaker one.

Perplexity: two bots, and no training crawler at all

Perplexity runs two agents, and unusually it claims to operate no training crawler whatsoever. Both bots are explicitly disclaimed from foundation-model training.

PerplexityBot is the AI-search index, “designed to surface and link websites in search results on Perplexity”. Perplexity states directly: “It is not used to crawl content for AI foundation models.”

Perplexity-User is the live fetch: “When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response.” Note that last clause. That’s a vendor describing the citation mechanism in its own documentation. Also explicitly not used “for web crawling or to collect content for training AI foundation models”.

Perplexity is the vendor most associated with actually passing clicks through, because it puts its sources front and centre. That’s our impression rather than something we measured, so treat it as a working assumption.

Like OpenAI, Perplexity says robots.txt doesn’t reliably bind its user-triggered fetcher: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” And like OpenAI, changes take time: “it may take up to 24 hours for our systems to reflect changes.”

Apple: the most carefully designed controls of anyone, and a trap most people walk into

Apple runs Applebot, which does two jobs at once. It builds the index behind Siri and Spotlight, and the same crawl feeds Apple’s foundation models. You’ll see it in your logs as a fairly ordinary-looking Safari user agent with Applebot/0.1 bolted on the end.

Then there’s Applebot-Extended, which is not a bot. Apple is explicit that it “does not crawl webpages”. It’s a control token you put in robots.txt to say “crawl me for search, but don’t train on me”. You will never see it in your logs, because it doesn’t exist as a thing that visits your site.

And there’s a third lever: the nosnippet tag, which lets Apple crawl you and index you but stops your content being used to generate an answer.

Three separate decisions. Crawl, train, ground. Apple is the only company on this page that gives you all three, and it’s worth saying plainly that this is what good looks like. Everyone else offers you fewer choices.

Now the trap. Apple’s own documentation says it: “If robots instructions don’t mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions.”

So if you blocked Googlebot on a staging area years ago and forgot about it, you’ve also blocked Apple without ever typing the word Apple. Conversely, if your robots.txt is a long list of rules aimed at Google, Apple is quietly obeying all of them. Worth ten minutes of your time to go and look.

One smaller thing: Applebot ignores the Crawl-delay directive, so if you’ve set one hoping to slow it down, it isn’t working. And iTMS, Apple’s podcast feed fetcher, doesn’t follow robots.txt at all.

Amazon: three bots, one of which won’t tell you what it does

Amazon runs Amazonbot, Amzn-SearchBot and Amzn-User.

Amzn-SearchBot is the index builder and it’s the clean one. Amazon states it isn’t used for training. It feeds Alexa’s answers.

Amzn-User is the live fetch. Real person, real question, right now. Amazon notes it “may not follow all robots.txt directives”, the same caveat OpenAI and Perplexity give.

Amazonbot is the awkward one. Amazon says it’s “used to improve our products and services” and “may be used to train Amazon AI models”. May. Not is, not isn’t. Amazon is the only company in this whole survey that won’t give a straight answer about whether its crawler feeds training, and it offers no separate training opt-out, so you can’t settle the question by configuration either. You either allow Amazonbot or you don’t.

Two more things about Amazon that catch people out.

It inherits your rules for other search engines. In Amazon’s words: “If robots.txt files don’t mention Amzn-SearchBot but allow other search bots, Amzn-SearchBot will crawl in accordance with the robots.txt directives given to other search bots.” Same pattern as Apple. Rules you wrote for Google are governing companies you never thought about.

Changes can take up to 30 days to land. Amazon may work from a cached copy of your robots.txt for a month. If you block something today and it’s still turning up next week, that’s documented behaviour, not defiance. Wait it out before escalating.

There’s also a small oddity worth knowing if you run an older site. Amazon has given the noarchive tag a meaning nobody else uses. Conventionally noarchive stops a search engine keeping a cached copy. Amazon reads it as “do not use the page for model training”. So a tag you set years ago for completely different reasons is now doing something at Amazon that you never intended.

Microsoft and Copilot: the one that surprises people most

Here is the finding that trips up nearly everyone who goes looking.

There is no Copilot crawler. Microsoft documents exactly five: Bingbot, AdIdxBot, BingPreview, MicrosoftPreview and BingVideoPreview. Not one of them is AI-specific, Copilot-specific or training-specific. There is no Microsoft equivalent of GPTBot or Google-Extended.

Copilot answers questions by grounding against the Bing index. The Bing index is built by Bingbot. Which means allowing Bing Search while blocking Copilot is not something you can express. There is no line of robots.txt that says it.

Every other company here gives you a dial. Microsoft gives you a switch.

For most businesses this resolves itself quickly, because almost nobody wants to disappear from Bing. But it’s worth understanding what you’re agreeing to, rather than assuming there’s a partial option sitting somewhere you haven’t found. There isn’t. Microsoft hasn’t published a rationale, a roadmap, or any acknowledgement that the gap exists.

A footnote for anyone reading their logs closely: the user-agent strings Microsoft publishes for BingPreview are identical to the ones it publishes for Bingbot. If that’s accurate, you can’t tell them apart in your logs and you can’t block one without the other. If it’s a documentation error, it’s been left standing on Microsoft’s own page.

Meta: five bots, and one of them you must not block

Meta has the largest fleet here, and the highest chance of an expensive mistake.

facebookexternalhit/1.1 is the preview fetcher. Every time somebody shares your link on Facebook, Instagram, Messenger or WhatsApp, this bot fetches the page to build the little card with your headline and image. Do not block this one. Block it and your links start appearing as bare grey URLs with no picture and no title, and your click-through falls off a cliff. Meta also notes it may bypass robots.txt for integrity checks, so blocking it may not even achieve what you wanted.

meta-externalagent/1.1 is the training crawler. This is Meta’s GPTBot equivalent.

meta-webindexer/1.1 is the AI-search index, and Meta is unusually direct about what it buys you: “Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI’s responses.”

meta-externalfetcher/1.1 is the live, user-triggered fetch, and Meta says it “may bypass robots.txt rules”.

meta-externalads/1.1 is advertising infrastructure.

Meta also recommends verifying its crawlers by IP address, then publishes no IP addresses. No ranges, no network identifiers, nothing. So facebookexternalhit is easy for anyone to impersonate and there’s no first-party way to catch them at it. That gap is Meta’s alone.

Common Crawl: the bot that isn’t an AI company

CCBot belongs to Common Crawl, a non-profit that maintains a free, open archive of the web. It isn’t an AI company. It doesn’t have a chatbot. It will never send you a visitor.

It matters because a great many AI developers train on the Common Crawl archive rather than crawling the web themselves. So one line about CCBot in your robots.txt reaches further than almost any other line you could write.

But blocking it is a door that only opens one way. It stops your content going into future crawls. It removes nothing from the archives already published, already downloaded, already sitting on other people’s servers. If your objection is about what happens from here on, blocking CCBot does something real. If you were hoping to take back what’s already out there, it doesn’t do that, and it’s better to know now.

ByteDance: we looked, and there’s nothing to tell you

You may also see ByteDance’s crawler, usually appearing as Bytespider. ByteDance is the only major operator on this list that publishes no crawler documentation at all: no confirmed user agent, no stated purpose, no published robots.txt policy. We checked. If you see it in your logs, treat it as an unknown rather than assuming it behaves like the others.

The mistake that actually costs money

Here’s where all of this becomes practical.

Most people who go looking into this subject arrive with one instinct: block the AI bots. They read something about scraping, decide they want out, and go and edit one file.

The problem is that at seven of the ten operators above, the company has split itself into separate bots with separate names, and the split isn’t obvious from the naming. GPTBot and ChatGPT-User both say ChatGPT. One will never send you a visitor. The other means a human being is asking a question right now and ChatGPT has come to your page for the answer.

Block both because they share a brand name and you’ve achieved the opposite of what you wanted. The training carries on at every other company you didn’t think of, and you’ve switched off the one bot that was bringing you customers.

So before you touch anything, work out which of these two sentences you actually agree with:

• “I don’t want AI companies training on my content.”
• “I don’t want AI assistants answering questions about my business.”

Almost everybody holds the first. Very few people hold the second, and the ones who do usually have a specific reason. They are different objections and they need different actions.

If it’s only training you object to, the bots to block are the training crawlers: GPTBot, ClaudeBot, meta-externalagent, Google-Extended, Applebot-Extended and CCBot. Leave everything else alone.

Never block a bot with User or SearchBot in its name unless you have specifically decided you don’t want to appear in AI answers at all. Those are the ones tied to citations and clicks.

Accept that Amazon and Microsoft are all or nothing, and make that call deliberately rather than assuming a middle option exists somewhere.

And don’t reach for a blanket User-agent: * block. It’s the one action that reliably delivers the exact opposite of what most people want.

Four other things that are easy to get wrong, all of them documented by the companies themselves.

Google’s opt-out is invisible, and that’s normal. Google-Extended has no user agent. Google’s words: “Google-Extended doesn’t have a separate HTTP request user agent string.” Set that opt-out and Googlebot keeps arriving at exactly the same rate, because the token governs what Google does with the data afterwards, not whether it collects it. People see Googlebot still coming and conclude the opt-out failed. It didn’t. That’s the design. The flip side is that you can’t confirm it from your logs either, because there’s nothing to look for.

Perplexity has no training crawler to opt out of. It says so on both its bots. Which means blocking Perplexity looks like close to pure downside: you lose the citations and the clicks that come with them, and you prevent no training, because by Perplexity’s own account none was happening. That’s our reading of their documentation rather than something we measured, but of everyone here, this is the block that makes least sense.

A robots.txt entry is a request, not a control. For the user-triggered bots, five of six companies say in writing that they may ignore it. Anthropic is the lone exception and claims its bots do respect robots.txt for user-initiated requests. So “block ChatGPT-User” is a guarantee at Anthropic and explicitly not a guarantee at OpenAI, Google, Perplexity, Amazon or Meta.

Blocking at your firewall is not a stronger version of blocking in robots.txt. It’s a different thing with different consequences, and it’s where most of the accidental damage happens.

Where to go from here

The most common version of this problem isn’t someone blocking the wrong bot on purpose. It’s a business that never blocked anything and still can’t work out why the AI assistants have nothing to say about them. Usually the answer is sitting in a robots.txt file written years ago, or in a security setting somebody switched on by default that nobody has looked at since.

Two companion pieces to this one. First, the two lines in your robots.txt that quietly delete you from Google - which is the request layer we mentioned above. Second, the network-edge piece on Cloudflare - which is where the enforcement happens, and where a single setting can override every carefully-tuned line in your robots.txt.