AI crawler statistics for small business websites: 40 numbers from 3,693 sites (2026)

Short answer: in October 2026 we checked 4,111 small business websites in 26 US cities and could analyze 3,693. 85.6% were fully open to AI crawlers. About 1 in 7 weren't, and only about 3% of sites block AI crawlers on purpose. The rest were stopped by bot checks and broken homepages. Every number from the study is below, with what it's out of.

The headline numbers

1. Sites sampled: 4,111 small and local business websites, 26 US cities and metros, 14 categories.
2. Sites analyzed: 3,693 (246 dead domains and 172 inconclusive ones excluded).
3. Fully open to AI crawlers: 3,163 of 3,693 (85.6%).
4. Not fully open: 530 of 3,693 (14.4%), about 1 in 7.
5. Of those 530, not caused by a robots.txt rule: 415 (78%).

What stopped AI crawlers (share of all 3,693)

6. Bot check shown to an identified crawler: 216 (5.8%).
7. Homepage didn't load: 183 (5.0%).
8. robots.txt blocks some AI crawlers: 86 (2.3%).
9. robots.txt blocks all major AI crawlers: 29 (0.8%).
10. robots.txt couldn't be read: 16 (0.4%).

robots.txt (share of the 3,370 sites with a readable robots.txt)

11. Block at least one of the 9 major AI crawlers: 109 (3.2%).
12. Block all 9: 29 (0.9%).
13. No robots.txt at all: 433 (12.8%); that's allowed by default.
14. Meta-ExternalAgent blocked: 107 (3.2%).
15. GPTBot blocked: 105 (3.1%).
16. Applebot-Extended blocked: 97 (2.9%).
17. ClaudeBot blocked: 96 (2.8%).
18. Google-Extended blocked: 79 (2.3%).
19. PerplexityBot blocked: 42 (1.2%).
20. OAI-SearchBot blocked: 34 (1.0%).
21. ChatGPT-User blocked: 33 (1.0%).
22. Claude-SearchBot blocked: 32 (0.9%).
23. Block GPTBot but allow OAI-SearchBot: 72 (2.1%).
24. Blocking sites that name AI crawlers rather than all bots: about 83%.
25. Wildcard rule blocking every crawler from the homepage: 1.2%.
26. Cloudflare's managed AI section in robots.txt: 30 (0.9%); about a quarter of all sites that block any AI crawler.
27. A Content-Signal line in robots.txt: 132 (3.9%).

Bot checks (share of all 3,693)

28. Cloudflare: 112 (3.0%).
29. SiteGround: 99 (2.7%).
30. Akamai and AWS WAF: 5 (0.1% each or less).

Homepages that didn't load (183 sites)

31. HTTPS certificate or TLS error: 140 (3.8% of all sites).
32. HTTP error from the server: 33.
33. Redirect loop: 6.

Cloudflare

34. Sites behind Cloudflare: 1,481 of 3,693 (40.1%).
35. Open to AI crawlers: 88.2% behind Cloudflare vs 84.0% not on Cloudflare.
36. Homepage didn't load: 0.6% behind Cloudflare vs 7.9% not on Cloudflare.

Website builders and structured data

37. Wix, Shopify, GoDaddy and Weebly sites open to AI crawlers, of those whose homepage loaded: 100% (233, 184, 153 and 80 sites).
38. Homepages with valid JSON-LD structured data: 1,830 of 3,283 (55.7%).
39. Valid JSON-LD on Squarespace homepages: 277 of 368 (75.3%); on Weebly: 5 of 80 (6.2%).

By type of business (share of each category's sites)

40. Most open: vets and pet services 90.8% (of 207). Least open: hotels and lodging 79.7% (of 202). Most bot checks: restaurants and bars 9.6% (of 342).

The full per-category table is in the study.

Being readable is the first step. To see whether AI actually names your business, who it names instead and where it contradicts your website, get the full $99 AI visibility audit, paid once.

How we measured

• Our crawler, PreferlyBot, identified itself, followed robots.txt and requested only robots.txt and the homepage, once per site, on Oct 22, 2026.
• "Bot check" means a challenge page instead of the homepage, as a new, clearly identified crawler sees it. Large AI crawlers may be treated differently.
• Sites that didn't respond, or redirected to plain http, were excluded, so the real "didn't load" share is likely a bit higher.
• Builders were detected only on homepages that loaded.
• Full method, charts and the downloadable data: How many small business websites block AI crawlers?

Data: Preferly, "How many small business websites block AI crawlers?" (2026), https://www.preferly.ai/research/ai-crawler-study. Licensed CC BY 4.0. Sample drawn from OpenStreetMap data, © OpenStreetMap contributors, ODbL.