Is Cloudflare killing your AI visibility?

Update, 15th September 2026: the date this article was written around has arrived. What follows still describes what was scheduled to change and where to look for it, but it is no longer a warning you can act on in advance. If you have not checked your Cloudflare settings, that check is now overdue rather than early.

On 15th September 2026, Cloudflare changed how it handles AI bots. Two separate things happened on that date, and they were not equally alarming. Most of the coverage you will see merges them, and the merged version got it wrong in both directions: it overstated the risk to brand new sites while badly understating the one facing yours.

So let me split them.

The half that is calmer than it sounds. New domains get a new default posture. Cloudflare’s own wording, from its documentation:

“On September 15, 2026, Cloudflare will set updated defaults for new domains: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed.”

Read “on pages that display ads” slowly. That is not a site-wide block. Cloudflare gives you three choices for each category of bot behaviour: block everywhere, block only on ad-bearing pages, or allow. The new default picks the middle one. If your site carries no display advertising, which covers most lead-generation sites and most e-commerce, then on Cloudflare’s own description of that default you are blocking nothing at all. AI search stays allowed either way.

So the frightening headline is probably not about you.

The half that is. On the same date, a setting you may already have ticked changed what it means. The legacy “Block AI bots” toggle, the one that has been sitting in Security Settings for a while now, used to carry an explicit carve-out. Cloudflare’s words, describing the setting as it stood before 15th September 2026:

“This setting blocks verified bots that are classified as crawling for the purpose of AI training, as well as a number of unverified bots that behave similarly. Note: This option excludes mixed-purpose bots that are used both for Training and for Search.”

That carve-out is what had been protecting your AI search visibility. From 15th September 2026 it read:

“Mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy ‘Block AI bots’ option.”

Nobody had to touch a setting for that to happen. If you, or a developer you worked with two years ago, ticked “Block AI bots” because training on your content felt wrong, you made a choice that deliberately spared the bots that also do search. On 15th September 2026 the definition widened underneath you, and the same tick now covers them.

That is the whole article in one mechanism. You asked to block training. You are now also blocking search, because some vendors run one crawler for both jobs, and a block beats an allow.

The one thing worth doing first

Open your Cloudflare dashboard, go to Security Settings, and look at Block AI bots.

Just look. You are finding out whether it is on, not changing it. Cloudflare’s own paragraph closed with a line that only made sense for people who already had a site: “Before September 15, all customers can opt out of these new defaults.” That opt-out window has now closed, but the check itself still matters. Most site owners genuinely don’t know what is switched on in their own dashboard.

If it is off, treat the rest of this as background reading. If it is on, you have a decision to review, and that review is now overdue.

Cloudflare is not being careless here

It would be easy to write this as a villain story. It is not one, and the reason matters, because it tells you what you can and cannot fix.

Cloudflare sorts bots by what they do rather than by who owns them:

“Rather than relying on a single ‘AI bot’ label, Cloudflare classifies bots by behavior - what a bot does on your site - so you can allow the behavior that helps your business and block the behavior that harms it. A single bot can have more than one behavior.”

Three of those behaviours are exposed as controls every customer can set, and Cloudflare defines them like this:

If you read the second article in this series, that split will look familiar. A content delivery network and eight competing AI companies reached the same taxonomy independently, which is decent evidence the distinction is real rather than marketing.

The problem lives in the last sentence of that quote: a single bot can have more than one behaviour. When one crawler does both search and training, Cloudflare has to pick a side, and it picks the safe one. Block wins. A site owner who wants the reasonable thing, index me but do not train on me, gets neither for that crawler.

One honest caveat, because this is where articles on the subject start inventing things. Cloudflare does not publish a list of which crawlers it treats as mixed-purpose, so nobody can name the ones you will lose. The only formulation that survives contact with the evidence is conditional: if a vendor uses one agent for both jobs, a training block takes out its search too, and you will not be notified.

Note what else sits in the new default. The Agent category covers bots fetching a page right now because a human just asked a question, which is the class that actually sends you referral traffic. On ad-bearing pages, it has been in the blocked column since 15th September 2026.

“My Cloudflare bot setting” is five settings

This is where most fixes go wrong. People change one thing, see no improvement, and conclude Cloudflare is broken. In fact Cloudflare ships five separate products that can stop an AI bot. They sit in different menus, come with different plans, and disagree with each other about whether you are allowed to write an exception.

Two things fall out of that table. The first is that “AI Crawl Control” used to be called “AI Audit”, so any guide written before the middle of last year will send you hunting for a menu item that no longer exists. The second is the exception column, which nobody mentions and which decides whether your fix works.

The free-plan trap

If you are on Cloudflare’s free plan with Bot Fight Mode switched on, and it challenges a bot you wanted, there’s no allow rule you can write. None. Cloudflare says so directly:

“You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules. This is because Bot Fight Mode does not run on the Ruleset Engine - it operates in a separate evaluation pipeline where Skip, Bypass, and Allow actions have no effect.”

Two further Cloudflare pages repeat it. Three documentation pages, one answer. So every “just add a WAF rule to allow it” reply you will find on a forum is wrong for free-plan Bot Fight Mode. Cloudflare’s own stated remedy is to change product:

“If you need to create exceptions for specific traffic (for example, your own API clients or monitoring tools), use Super Bot Fight Mode instead. Super Bot Fight Mode runs on the Ruleset Engine and supports Skip rules.”

Super Bot Fight Mode starts at the Pro plan. So your two options on Free are to switch Bot Fight Mode off entirely, or to pay for the tier that lets you be precise. That is a commercial fact wearing a technical costume, and it is worth naming as one. The cheapest bot protection available is also the bluntest, and blunt is the wrong property when you are trying to keep one category of visitor while excluding another.

There is one lever that runs before Bot Fight Mode, an IP Access rule. We wouldn’t build an AI allowlist on it. The vendors’ published IP lists decay badly, and one file examined for this series was dated February 2025 and listed eight address ranges. Pin your allow rule to something like that and you risk blocking the exact crawler you wrote the rule to permit.

The rule that explains all of it: the blunt layer runs first

Everything above becomes predictable once you know the order things run in. Cloudflare publishes it:

Traffic → WAF custom rules (including AI Crawl Control crawler blocks) → Cloudflare Bot Solutions → AI Crawl Control: Pay Per Crawl

Bot Fight Mode does not appear on that line at all, because it runs in its own pipeline outside the rules engine. Your careful decision about one named crawler sits downstream of a switch that has already made a cruder one and cannot be argued with.

Which produces the most useful troubleshooting paragraph Cloudflare has written on the subject:

“If you have set certain AI crawlers to Allow in AI Crawl Control, but they are still being blocked, check for upstream WAF custom rules that may be blocking them. Since the AI Crawl Control rule only includes blocked bots, allowed bots may still be affected by other security rules that execute before the AI Crawl Control rule. These upstream rules will affect traffic but may not be visible in AI Crawl Control analytics.”

Setting a crawler to Allow can look successful on screen while the bot is still being blocked somewhere the screen does not show you. The mechanism is almost elegant: when you block a crawler in AI Crawl Control, Cloudflare writes a WAF custom rule to enforce it. When you allow one, it writes nothing. Allow is the absence of a rule, which is precisely why it cannot overrule anything that ran earlier.

It runs in the other direction too. Cloudflare appends its AI Crawl Control rule at the end of your existing custom rules, and warns that a Skip, Redirect or Transform rule you wrote previously “may allow bots to bypass the block”. Its advice is to move the AI Crawl Control rule to the top.

So a decision you make in the AI dashboard, in either direction, can be silently overridden by a rule you wrote eighteen months ago for an unrelated reason, and the AI dashboard will not tell you. If you take one operating principle from this piece, take that one.

Verified does not mean allowed

There is a common and expensive assumption that if a bot is on Cloudflare’s verified list, it gets through. It doesn’t.

Cloudflare’s definition of a Verified bot is a good one. It is a bot that is “transparent about who it is and what it does: it represents itself honestly and does not abuse the access that honesty earns”. To qualify, a crawler has to prove its identity deterministically, by cryptographic signature, published IP list or reverse DNS, and then it has to behave itself, which means obeying robots.txt, keeping its request rates sane and never having been caught dodging what a site owner asked for.

Then read what that status actually buys:

“Each blocking option will block Verified bots classified with that behavior, plus additional unverified bots that fall under these classifications.”

Verified status is an identity claim, not an access pass. A signed, impeccably behaved crawler gets no protection from your settings whatsoever. It just becomes identifiable enough to be blocked accurately. Good behaviour buys a vendor visibility on your site, not access to it, and “they’re verified, so we’re fine” is not a sentence anyone should say out loud.

We cannot give you a list of which AI bots hold verified status, and the reason is a decent illustration of the whole problem. Cloudflare publishes the directory through Cloudflare Radar. We tried four times on 14 September, two ways. Two URLs returned 403, the API wanted authentication we don’t have, and a full browser hit a Cloudflare managed challenge that sat on “Performing security verification” and never cleared. The public directory of bots Cloudflare has certified as honest sits behind a Cloudflare bot challenge, and the challenge fires before the question of who is asking can be considered at all.

There is also a documented tension worth knowing. Cloudflare lists, among its grounds for removing a service from the verified allowlist, “An AI Crawler that does not respect the crawl-delay directive in robots.txt.” Three separate AI vendors state plainly in their own documentation that they do not support crawl-delay. Both positions are published and they cannot both be fully operative.

Go and look now, because you cannot look backwards

Cloudflare gives you four views, and they answer different questions. Use them in this order.

Security Events is the one that ends arguments, because its “Events by service” breakdown names the product responsible. That is how you discover it was Bot Fight Mode doing it, not the WAF rule you have spent an afternoon rewriting.

Two limits to know before you trust any of this. On the free plan, AI Crawl Control identifies crawlers by their user agent string, so your crawler list is really a list of things that claimed to be AI crawlers. The honest ones show up correctly, and anything dishonest is listed under whatever name it fancied typing that day. Cloudflare is upfront about this and offers better detection on paid plans.

The second limit should move you. Security Events keeps 24 hours of history on Free and Pro, three days on Business, thirty on Enterprise. If a crawler stopped visiting a fortnight ago because a setting changed, that evidence no longer exists. You cannot audit backwards, only start watching, which is an argument for opening the dashboard now rather than after your traffic reports look strange.

The bridge back to your robots.txt

The previous article in this series was about robots.txt, a polite request a crawler can read and choose to honour. This one is about the layer underneath, where a bot is stopped at the network and never gets to read your robots.txt at all. The two layers must agree, or the polite one is wasted.

Cloudflare ships a screen that measures exactly that failure. The Directives tab shows the HTTP status returned when crawlers ping your robots.txt, and Cloudflare’s remediation advice reads: “If the Status is 404 Not Found, create a robots.txt file to provide clear directives. If the file exists, check for upstream WAF rules or other security settings that may be blocking access.”

One methodology note, so you don’t raise a false alarm. The violations table compares your current robots.txt against past requests, so adding a new Disallow line makes every historic request to that path appear as a violation, even though it was not one at the time.

Pay per crawl, and why we are not telling you to use it

You will read a lot about Cloudflare’s pay-per-crawl model, where a crawler either presents payment intent or receives an HTTP 402. Interesting direction. It is also still in closed beta, as of Cloudflare documentation dated 28 July 2026, so it isn’t something a UK SME can switch on this week.

It obeys the same ordering rule as everything else, and Cloudflare says so in two separate places: if your WAF or bot settings have already blocked a crawler, it never reaches the point where it could have paid you. Blocking gets there first, and first wins. You cannot monetise traffic you have already turned away.

What to do now

Three steps, in order.

One. Open Security Settings and look at Block AI bots. You are establishing what is currently on, not changing anything yet. If it is on, you now know your AI search exposure widened on 15th September 2026.

Two. Open AI Crawl Control and look at the Crawlers tab. Find out which AI crawlers are actually reaching your site and what action is set against each. Note the free-plan caveat above when you read it.

Three. Decide what you actually want, then set it in one place. For most businesses we work with the honest answer is: allow search, allow the live user-triggered fetchers, and make a deliberate decision about training rather than inheriting one. Then check Security Events to confirm nothing upstream is quietly overriding you.

None of this is Cloudflare’s fault, and it is certainly not a reason to stop using Cloudflare. It is a very good product doing exactly what it was asked to do. The trouble is that it was asked a long time ago, probably by someone who is no longer around, using a word whose definition changed on 15th September 2026.

For the wider picture around this piece, including the robots.txt and bot-identification articles alongside it, see the running index of this series.

If you would rather someone went and looked for you, that is what our GEO service does. Get in touch and we will audit what your current settings are actually doing to your AI visibility.