Two lines that delete you from Google: the robots.txt file for AI visibility

Open your robots.txt file. It sits at yourdomain.co.uk/robots.txt, it is a plain text file, and you can read it in a browser right now without logging into anything.

If you find this in it, stop reading and go and fix it:

User-agent: *
Disallow: /

Those two lines remove you from Google, Bing, every AI answer engine and every link preview at the same time. Not just the AI companies you meant to block.

Most people with those two lines put them there deliberately, to stop AI companies taking their content for training. The instinct was sound. The file was not. robots.txt only binds operators who fetch it and choose to honour it, so the blanket block does not reliably stop training either. It costs you everything and buys you very little.

You can have the opt-out without the blackout. This follows on from Not all AI bots are created equal: the forty-odd AI bots hitting your site are doing four different jobs, and the ones that send you visitors are not the ones most people block. This is the file you write once you know that.

The file

This is our recommended default for a business that wants to be found in AI answers. Copy it, then read the notes underneath, because one group in it is deliberately wrong until you fix it.

# ----------------------------------------------------------------------
# robots.txt : maximum AI visibility
# Site: www.example.com
# Last reviewed: 2026-09-14
# ----------------------------------------------------------------------

# 1. BASELINE FOR EVERY CRAWLER
#    The two Disallow lines below are PLACEHOLDERS. Replace them with your
#    own paths, or delete them if you have nothing that needs hiding.
#    What belongs here: admin login, basket, checkout, account pages,
#    internal search results. Nothing else, unless you can defend it.
#    Keep this list as short as you can defend. Anything disallowed here
#    is also invisible to every AI answer engine.
#    Your stack's starting point is in PLATFORM BASELINES, below.
#    Whatever you put here, repeat it in group 2. See note 2.
User-agent: *
Disallow: /REPLACE-ME-admin/
Disallow: /REPLACE-ME-basket/

# 2. AI-SEARCH INDEX BOTS. These decide whether you are eligible to be
#    cited at all. Named explicitly because Applebot and Amzn-SearchBot
#    otherwise inherit whatever rules you wrote for Googlebot.
User-agent: Googlebot
User-agent: bingbot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Applebot
User-agent: Amzn-SearchBot
User-agent: meta-webindexer
#    The next two lines are NOT a duplication error. They are the same
#    rules as group 1, repeated, because this group does not inherit them.
#    If you edit group 1 and not this, you have just opened those paths to
#    every bot named above. Keep the two lists identical.
Disallow: /REPLACE-ME-admin/
Disallow: /REPLACE-ME-basket/

# 3. LIVE USER-TRIGGERED FETCHERS. A human is waiting on an answer right
#    now. These are the hits that correlate with citations and clicks.
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Amzn-User
User-agent: Google-GeminiNotebook
User-agent: Google-Agent
User-agent: meta-externalfetcher
Allow: /

# 4. LINK PREVIEWS AND SOCIAL UNFURLS. Blocking these breaks your share
#    cards and gains you nothing on the AI front.
User-agent: facebookexternalhit
Allow: /

# 5. TRAINING CRAWLERS AND TRAINING CONTROL TOKENS. Allowed in this
#    variant. Listed explicitly so the decision is on the record rather
#    than an accident of omission. See Variant B to reverse it.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Three things to know before you use it.

Group 1 is a placeholder and it is meant to be edited. Groups 2 to 5 are not. The REPLACE-ME paths are fake on purpose, so nobody can paste this and believe group 1 is already right. Groups 2 to 5 are about bots rather than platforms, so they are identical on every stack.

Groups 4 and 5 change nothing on a site with no existing blocks. robots.txt defaults to allow, so an absent rule is already permission. They are there so the next person to open the file can see the decision was deliberate.

Review the date in the comment, not just the rules. Tokens change. Google renamed Google-NotebookLM to Google-GeminiNotebook, with the old name supported only until August 2026.

The rule that quietly breaks well-meaning files

Look at groups 1 and 2 again. The same two Disallow lines appear twice. That looks like sloppy copy-paste. It is not, and understanding why is the single most useful thing in this article.

robots.txt is not a set of layers. It is a set of separate instruction sheets. Each bot picks up the one sheet with its name on it and never reads the others. Anything you wrote on the general sheet has to be copied onto the named one too.

Google’s specification says it three times over:

“Only one group is valid for a particular crawler.”

“Other groups are ignored.”

“User agent specific groups and global groups (*) are not combined.”

RFC 9309, the published standard for robots.txt, comes at it from the other side, in section 2.2.1:

“If no matching group exists, crawlers MUST obey the group with a user-agent line with the ‘*’ value, if present.”

So User-agent: * is a fallback, not a foundation. The moment you write User-agent: GPTBot anywhere in the file, GPTBot stops reading your * group entirely, and every Disallow you carefully put there becomes, for that bot, permission.

We see it go wrong regularly. Somebody adds a friendly User-agent: GPTBot group with Allow: / under it, to a file that already disallows /wp-admin/, /checkout/ and /my-account/. They have just handed GPTBot all three. The file looks more thoughtful than it did yesterday. It is leaking.

The fix: repeat your baseline disallows inside every named group, and change both together. One piece of good news: multiple groups naming the same agent do get merged, so you cannot break it by splitting one bot’s rules across two places.

Filling in group 1, without guessing

Group 1 is the only part that depends on what you built your site with. Most published advice quietly assumes WordPress. First question, and it decides whether any of this is usable: can you even edit the file?

Three of those need saying out loud.

Squarespace. You cannot do any of this. Their own Help Center: “All Squarespace sites use the same robots.txt file and Squarespace users can’t access or edit the file.” There is a settings panel with bulk toggles, but one “block AI crawlers” switch cannot separate AI-search visibility from model training, which is the entire point of the file above.

Shopify. Do not write group 1 yourself. Shopify generates the defaults and, in its words, they are “updated regularly to ensure that SEO best practices are always applied”. Add a robots.txt.liquid template, output those defaults through the provided Liquid objects, then append groups 2 to 5.

WordPress. Two lines, and the second is not optional:

Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

admin-ajax.php sits inside the admin directory but is the endpoint many plugins use for front-end functionality. Block the directory without that Allow and you degrade how crawlers render your public pages. Most sites want a third line, to keep internal search results out:

Disallow: /*?s=

WooCommerce, on top of the WordPress lines:

Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?add-to-cart=

Those slugs are configurable, and plenty of UK stores rename /cart/ to /basket/. Check the live site before you paste, or you will disallow three paths that do not exist while the real basket stays open and the file looks correct.

Static and bespoke builds. Group 1 may legitimately be empty. No admin on the public hostname and no basket means nothing to disallow.

Not in the table? Do not adapt the nearest-looking entry, because a confident wrong path fails silently. Derive it: fetch what is already there with curl -sS https://www.example.com/robots.txt and start from it rather than over it; log in, and whatever URL your admin sits at is your first Disallow; add to basket and start a checkout, and note both URLs; run a site search, and the query parameter gives you the Disallow: /*?param= line. Then stop, because every line you add here is content that becomes ineligible to be cited in an AI answer.

Copy the finished list into group 2 as well, unchanged.

Three variants, and the question that picks between them

The syntax is the easy part. Before you touch anything, answer one question honestly: what do you actually object to?

“AI training on my content” and “AI answering questions about my business” are different objections. Most businesses hold the first one and not the second. Seven of the ten operators we surveyed have split themselves into separately controllable bots, and the split is not obvious from the names.

Variant A: maximum visibility, training accepted. The file above, unchanged. Your content may train foundation models at OpenAI, Anthropic, Google, Apple, Meta and Amazon, and anyone training from the Common Crawl corpus. No attribution, no payment. Offered as a view rather than a finding: for most SMEs that is still the right trade, because obscurity is the bigger commercial threat.

Variant B: visible in AI answers, not used for training. Keep groups 1 to 4 exactly as they are. Replace group 5 with this:

# 5. TRAINING CRAWLERS: BLOCKED. These fetch to build training corpora.
#    None of them will ever send you a visitor.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

# 5b. TRAINING CONTROL TOKENS: BLOCKED. Neither of these is a bot. They
#     are switches that govern what the vendor does with data it has
#     already collected, so blocking them does NOT reduce your crawl
#     traffic and will NOT show up in your logs.
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# 5c. AMAZON. Amazonbot is the bundled crawler: Amazon says it "may be
#     used to train Amazon AI models" and offers no way to separate that
#     from its other uses. Blocked here. Amzn-SearchBot and Amzn-User,
#     which both explicitly disclaim training, stay allowed in groups 2
#     and 3 above.
User-agent: Amazonbot
Disallow: /

And one page-level tag, because Amazon’s training opt-out is not in robots.txt at all:

<meta name="robots" content="noarchive">

That tag normally suppresses a cached copy. Amazon assigns it a different meaning, in their own words: “noarchive (do not use the page for model training)”. It is the only training lever Amazon gives you, which is why a tag belongs in an article about a text file.

Variant B works cleanly at OpenAI, Anthropic, Meta and Apple, and costs nothing on the visibility side. It works at Google too, though you cannot audit it.

And it does not work at Microsoft at all. Microsoft publishes no AI-specific control token. Copilot grounds against the Bing index, and the Bing index is built by bingbot. Keep bingbot and you accept Copilot. Block bingbot and you leave Bing organic search, which almost no UK business should do. Anyone telling you Variant B is airtight has not checked.

Variant C: publisher blocking training hard. For a business whose content is the product: a news site, a paid research operation, a subscription archive. Groups 1 to 4 stay:

# 5. EVERY DOCUMENTED TRAINING CRAWLER AND CONTROL TOKEN: BLOCKED
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Amazonbot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Plus <meta name="robots" content="noarchive"> on every page you are protecting.

Then stop, because the next step is not robots.txt. A publisher with real commercial exposure needs enforcement at the network edge, not a politely worded request file.

Variant C costs nothing on the AI-search side, since every retrieval and citation bot in groups 2 and 3 stays allowed. What it does not do is remove anything. Content already crawled, already in a training set, already sitting in a published Common Crawl archive is untouched by every line in that block. If the goal is retraction, robots.txt is the wrong tool.

The ways this goes wrong, ranked by what it costs you

Ranked by commercial damage, not by how often they happen. The first three are what we check before anything else on a client site.

1. The blanket block. Covered at the top. Catastrophic, and it is the default instinct. The fix is Variant B.

2. Blocking the retrieval bots and leaving the training crawler running. The exact inverse of the goal, and it happens because the names share brand words. GPTBot and ChatGPT-User both say ChatGPT. Somebody searches the vendor name, blocks everything that matches, and stops. But ChatGPT-User, Claude-User, Perplexity-User and Amzn-User mean a human is waiting on an answer right now. Never block a token containing User or SearchBot.

3. Allowing GPTBot and believing the job is done. The most widespread bad advice in the field, and some of it was ours. GPTBot is OpenAI’s training crawler. It does not build the index ChatGPT searches and it does not fetch pages to answer questions. The two that decide whether you can be cited are OAI-SearchBot and ChatGPT-User. Allow GPTBot alone and you have optimised for the one OpenAI bot with no visibility value at all.

4. The named AI group that leaks admin and basket URLs. The group precedence problem above. On an e-commerce site this can surface basket and account URLs in AI answers.

5. The misspelled or non-existent token. User-agent: ChatGPT, User-agent: Claude, User-agent: Google*. The first two are brand names, not product tokens, and matching is on the whole token rather than a prefix, so ChatGPT does not match ChatGPT-User. The last is the wildcard trap: wildcards do not work in the User-agent line at all. Google’s specification is blunt, that “All non-matching text is ignored (for example, both googlebot/1.2 and googlebot* are equivalent to googlebot)”. So Google* reduces to Google, which is nobody.

A wrong token creates a group no crawler matches, and the bot you meant to address falls through to your * group instead. Nothing errors and nothing warns you. Take every token from a first-party vendor page, never from a listicle or a chatbot.

One token not to delete on that basis: anthropic-ai. It is not in Anthropic’s current documentation, which names ClaudeBot, Claude-User and Claude-SearchBot and nothing else. But we have seen the string in our own server logs in the past week, so a rule naming it is matching live requests. “I cannot find it in the vendor’s documentation” and “it does not exist” are different findings, and a documentation check only establishes the first. Leave it in an existing file. What we cannot tell you is who operates it or whether it reads robots.txt at all, so do not count it as a working control. For Anthropic training, the control resting on published documentation is ClaudeBot.

6. IP-blocking the bot instead of using robots.txt. Anthropic states the mechanism outright: “Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.” Firewall a bot at the IP layer and it can never fetch your robots.txt, so it can never learn you wanted to opt out.

7. Not naming Apple and Amazon, so they silently follow your Google rules. Apple: “If robots instructions don’t mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions.” Amazon: “If robots.txt files don’t mention Amzn-SearchBot but allow other search bots, Amzn-SearchBot will crawl in accordance with the robots.txt directives given to other search bots.” Silent inheritance cuts both ways, and either way you did not decide it. Which is why both are named in group 2 above.

8. Blocking JavaScript and CSS. Applebot renders both, and blocking them breaks Apple’s rendering exactly as it breaks Google’s. You get indexed as a worse version of yourself, and the traffic looks normal, so it is harder to spot than a block. Never disallow /wp-content/, /assets/, .js or .css wholesale.

9. The subdomain with no robots.txt. Rules are read per host, not per domain. www.example.com/robots.txt governs www and nothing else. Your shop., blog. and staging. hosts are wide open, and they are often the ones carrying baskets and account pages.

10. Deciding the Google training opt-out failed because Googlebot is still crawling. It has not failed. Google is explicit: “Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.” Disallowing it will not reduce your crawl traffic by a single request, and that is the design working. Same for Applebot-Extended. Worth knowing before you look at the logs, because the usual outcome is somebody reversing a correct configuration.

11. Your robots.txt returning a 500 during a deploy. RFC 9309 says a 5xx means assume complete disallow. Amazon’s published wording says an unfetchable file means behave as though it does not exist, which is full access. You cannot assume the safe reading applies. For a Variant C publisher, a twenty-minute deploy window is, at some vendors, twenty minutes of unrestricted crawling. Monitor /robots.txt for a 200 as an uptime check.

What robots.txt cannot do

A business that believes more of this file than is true makes worse decisions than one that knows the limits.

It is a request, not a control. Nothing in robots.txt is enforced. It is a sign on the door, not a lock on it. Every claim in this article about what a bot will do is a claim about what a vendor has said it will do.

It does not bind user-triggered fetches at five of six vendors. The biggest limit, and they document it themselves.

So “block ChatGPT-User” is not a guarantee at OpenAI and is a guarantee at Anthropic, by their own statements. Google goes furthest, publishing a three-way taxonomy of its own compliance: common crawlers “always respect robots.txt rules for automatic crawls”, special-case crawlers “may or may not respect robots.txt rules”, and user-triggered fetchers “ignore robots.txt rules”. That middle category is the most honest sentence any vendor has published on this.

It cannot retract anything. Every control here is forward-looking at every vendor. CCBot is the clearest case: blocking it keeps you out of future crawls and removes nothing from archives already published and already downloaded by other people.

It cannot bind an operator that publishes no token. ByteDance publishes no crawler documentation at all. You cannot write a rule for a token nobody will confirm, and if you write one you cannot tell whether it worked.

It cannot be audited for control tokens with no user agent. Google-Extended and Applebot-Extended produce no log line by design. You are trusting a stated policy, which is reasonable, and a different class of assurance from a block you can watch working.

It does not protect anything, and it advertises what you named. robots.txt governs crawling. A disallowed URL is still publicly reachable by anyone who types it, and robots.txt is a public file listing the paths you consider sensitive. Never use it to hide something. If it must not be reached, put authentication on it. If it must not appear in results, use noindex, which is a page-level directive and a different mechanism. And do not list sensitive paths, because Disallow: /internal-pricing-2026/ is a signpost.

It has no effect on a bot that never fetches it. A bot blocked at your CDN or firewall never reads your robots.txt at all, so the two layers have to agree or the polite one is wasted. That is the next piece, and it is where the Cloudflare settings live.

Checking it worked

Saving the file proves nothing. What you need to know is whether the change is live, and whether anything is obeying it. Check each host separately, not each domain:

curl -sSI https://www.example.com/robots.txt
curl -sS  https://www.example.com/robots.txt

You want a 200, text/plain, no redirects, under 500 KiB, and byte content matching what you wrote, which catches smart quotes and a CMS helpfully tidying your file. Then test the intent by hand. Google Search Console has a robots.txt report; nothing equivalent exists for OpenAI or Anthropic. You will not catch a leaking group by reading the file, because everyone reads their own file and sees what they meant.

Then wait before concluding anything. Most vendors document roughly 24 hours of cache and Amazon documents up to 30 days, so a month of unchanged Amazon behaviour is correct rather than a compliance failure.

What we could not establish

Everything above comes from vendor documentation fetched on 14 September 2026, from RFC 9309, from Google’s robots.txt specification, or from our own server logs. A confident wrong answer in this subject fails silently and looks correct, so here are the gaps.

• Who operates anthropic-ai, and whether it reads robots.txt at all. The string is in our logs, which proves something is presenting that name. It does not establish who is behind it, and we did not test whether it fetches the file. Unverified in both directions.
• Whether AI crawlers other than Googlebot support the * and $ path wildcards. Both are in RFC 9309, and Google states support for Googlebot. No vendor states it for OAI-SearchBot, ClaudeBot, PerplexityBot, Amzn-SearchBot or meta-webindexer. Hence the advice above: clever patterns in your * group, boring AI groups.
• Squarespace’s AI crawler toggles. The non-editability quote is first-party. The description of the toggle panel is secondary reporting we did not confirm.
• Shopify’s default disallowed paths, and Wix’s. Neither is enumerated on the vendor page we fetched. Read the live file on the site in front of you.
• Magento and Adobe Commerce. The Adobe documentation URL returned a 404 and we found no first-party replacement, so there is no path list from us. Same for BigCommerce, Webflow, Ghost, Craft, Umbraco, Sitecore and Salesforce Commerce Cloud.
• Bytespider. Widely cited, and ByteDance publishes no first-party documentation for it whatsoever. A verified negative, so it is deliberately excluded from every block above. If its volume is costing you money, that is a CDN decision.
• claude-web. Not in Anthropic’s current documentation, and unlike anthropic-ai we have not seen it in our logs. Harmless to leave in an existing file. Do not add it to a new one.
• Whether Google-NotebookLM is honoured as an alias for Google-GeminiNotebook. Google documents the old name as supported only until August 2026 and does not say whether it is an alias.
• BingPreview as a separately controllable agent. Microsoft publishes user-agent strings for it that are byte-identical to bingbot. Do not write rules that depend on separating the two.
• What Amazonbot actually contributes. Amazon will not say beyond “improve our products and services”, so blocking it means giving up an unquantified benefit.
• A first-party citation for the crawl versus index distinction. Standard technical SEO and we are confident in it, but stated above as our reading rather than a quotation.
• File size limits outside Google. Google enforces a 500 KiB ceiling. No other vendor in our survey publishes one, and no published limit is not the same as no limit.

Where this leaves you

Three things, in order. Fetch your robots.txt and check it does not contain the two lines at the top of this article. Work out whether your objection is to training or to being answered about. Then write group 1 from the site actually in front of you rather than from anyone’s template, and copy it into group 2.

That covers the request layer. The enforcement layer sits at your CDN, where a single bot-fighting setting can override every line you just wrote. That is where this series goes next.