Is Cloudflare blocking AI crawlers from my website?
It might be. Cloudflare can block AI crawlers through its Search, Training and Agent settings, publish a no-training preference in your robots.txt, or show a bot check to automated visitors. Since September 15, 2026, new domains get new defaults that block some AI crawlers. Check your robots.txt and your Cloudflare settings.
Updated October 22, 2026
Three ways Cloudflare can turn AI away
- AI crawler settings. Separate controls for Search, Agent and Training crawlers, each set to allow, block on pages with ads, or block on all pages. A block refuses the request at Cloudflare’s network, whatever robots.txt says.
- A robots.txt preference. Cloudflare can add a section to your robots.txt (“BEGIN Cloudflare Managed content”), now generated from your Search, Agent and Training settings, with Content Signals that state your preference. On its own it records your choice; it doesn’t block anything.
- Bot Fight Mode and challenges. A check page (“Just a moment...”) shown to automated visitors instead of your site.
The September 2026 change
Cloudflare replaced its single “Block AI bots” switch with the three controls above. Since September 15, 2026, new domains start with Search allowed and Training and Agent crawlers blocked on pages that show ads. A setting that blocks AI training can also block crawlers used for both search and training; Cloudflare’s “Disallow AI Training” option refuses training and keeps you in search.
How to tell from the outside
Open yoursite.com/robots.txt: a “BEGIN Cloudflare Managed content” block means the preference is on. Our free crawler check reports a Cloudflare bot check if your homepage answers with one.
How to change it
In the Cloudflare dashboard, open your domain, go to Security, then Settings, and find the AI bot policies. Allow the Search and Agent crawlers if you want AI answers to use your site. Bot Fight Mode is in the same settings, under bot traffic.
Your web host may use Cloudflare for you
Many site builders and hosts run their sites through Cloudflare, and the setting may be theirs, not yours. Ask their support.
Other bot-check vendors
Akamai, DataDome, HUMAN (PerimeterX), Imperva, Sucuri, SiteGround and AWS WAF work in a similar way: a check page for automated visitors, set by your host or your security settings.
Questions
Does Cloudflare block Googlebot?
Not by default. Search engine crawlers such as Googlebot are on Cloudflare’s verified bots list. But a setting that blocks AI training can also block crawlers used for both search and training. To keep search, use Disallow AI Training instead.
I didn’t change anything. Why is AI blocked?
A new domain’s defaults, a plan change, or your host’s settings can add AI blocks. Check robots.txt and the bot settings.
What are Content Signals?
A Cloudflare-proposed line in robots.txt that states whether content may be used for search, AI input or AI training. It records your preference. It doesn’t block anything.
Will allowing AI crawlers slow my site?
AI crawlers fetch far fewer pages from a small business site than search engines do. You can allow the search and user crawlers and still block training ones.
Sources
No email. It reads your robots.txt and homepage, the way AI crawlers do.