40% of small business sites run behind Cloudflare. Are they more or less open to AI?
Short answer: 1,481 of the 3,693 small business sites we checked (40.1%) were behind Cloudflare. They were more likely to show a bot check, but much less likely to have a broken homepage, so overall more of them were open to AI crawlers: 88.2%, against 84.0% for sites not on Cloudflare.
Side by side
| Result | Behind Cloudflare (1,481) | Not on Cloudflare (2,212) |
|---|---|---|
| Open to AI crawlers | 1,306 (88.2%) | 1,857 (84.0%) |
| Bot check | 112 (7.6%) | 104 (4.7%) |
| Homepage didn't load | 9 (0.6%) | 174 (7.9%) |
| Blocks some AI crawlers | 44 (3.0%) | 42 (1.9%) |
| Blocks all major AI crawlers | 2 (0.1%) | 27 (1.2%) |
| robots.txt unreadable | 8 (0.5%) | 8 (0.4%) |
What's going on
- More bot checks. Cloudflare's bot protection is the single most common source of bot checks in our study. A setting meant to stop scrapers also stops some honest crawlers.
- Far fewer broken homepages. Cloudflare handles HTTPS certificates for the sites it fronts, so the certificate errors that took down 140 homepages elsewhere barely show up.
- Partial blocks, not total ones. Cloudflare sites were more likely to block some AI crawlers and much less likely to block all of them. 30 sites carried Cloudflare's managed robots.txt section, which adds AI-crawler rules for you.
What to check in your Cloudflare dashboard
- Security settings: is "Bot Fight Mode" or a challenge for AI bots turned on?
- AI crawler controls: is "block AI bots" or the managed robots.txt on, and is that what you want?
- Then run the free AI crawler check to see what an identified crawler gets.
Our guide Is Cloudflare blocking AI crawlers? has the step by step.
Being readable is the first step. To see whether AI actually names your business, who it names instead and where it contradicts your website, get the full $99 AI visibility audit, paid once.
Data: Preferly, "How many small business websites block AI crawlers?" (2026), https://www.preferly.ai/research/ai-crawler-study. We checked 4,111 small and local business websites in 26 US cities on Oct 22, 2026 and analyzed the 3,693 that responded. The aggregate data is free to reuse under CC BY 4.0 with that credit.
Questions
Does Cloudflare block AI crawlers by default?
Cloudflare offers settings that block or challenge AI crawlers, and their defaults have changed over time. In our sample, 112 Cloudflare sites (7.6%) showed a bot check and 30 had Cloudflare's managed AI rules in robots.txt. Check your own dashboard.
Should I stop using Cloudflare?
No. In our data Cloudflare sites were more open to AI overall, mostly because their HTTPS worked. Adjust the bot settings instead.
3 AI models, 5 customer questions, about a minute. Or check whether AI crawlers can reach your site.
No email. It reads your robots.txt and homepage, the way AI crawlers do.