Only 3.2% of small business sites block AI crawlers in robots.txt. Here's who they block

Short answer: deliberate blocking is rare. Of the 3,370 small business sites where we could read robots.txt, 109 (3.2%) block at least one of the nine major AI crawlers from the homepage, and 29 (0.9%) block all nine.

The crawlers that get blocked most

Share of the 3,370 sites with a readable robots.txt:

Crawler	Who runs it	What it does	Sites blocking	Share
Meta-ExternalAgent	Meta	Training	107	3.2%
GPTBot	OpenAI	Training	105	3.1%
Applebot-Extended	Apple	Training	97	2.9%
ClaudeBot	Anthropic	Training	96	2.8%
Google-Extended	Google	Training (Gemini)	79	2.3%
Claude-User	Anthropic	Fetches pages for a user	43	1.3%
PerplexityBot	Perplexity	Search index	42	1.2%
OAI-SearchBot	OpenAI	ChatGPT search	34	1.0%
ChatGPT-User	OpenAI	Fetches pages for a user	33	1.0%
Claude-SearchBot	Anthropic	Search	32	0.9%

The pattern is clear: training crawlers get blocked about three times as often as the crawlers that fetch pages for live answers.

Most blocks name AI crawlers on purpose

When a site blocks, it's usually with rules that name specific AI crawlers, not a blanket "no bots" rule: about 83% of blocking sites did it that way. Only 1.2% of readable robots.txt files had a wildcard rule that blocks every crawler from the homepage.

Is blocking the right call for you?

For most local businesses, being found matters more than keeping your text out of training data. If you do want to opt out of training, you can block the training crawlers and still let the search and user crawlers in, so AI can look you up when a customer asks. Our guide to whether to block AI crawlers goes through the trade-off, and this one has the exact lines.

Data: Preferly, "How many small business websites block AI crawlers?" (2026), https://www.preferly.ai/research/ai-crawler-study. We checked 4,111 small and local business websites in 26 US cities on Oct 22, 2026 and analyzed the 3,693 that responded. The aggregate data is free to reuse under CC BY 4.0 with that credit.