1 in 7 small business websites aren't fully open to AI. Is yours one of them?
Short answer: about 1 in 7. We checked 3,693 small business websites in 26 US cities, and 530 of them (14.4%) weren't fully open to AI crawlers. Only 115 of those had a rule in robots.txt aimed at AI. The rest were bot checks, homepages that didn't load, and robots.txt files we couldn't read.
That's the part that surprised us. When people talk about websites "blocking AI," they usually mean a deliberate choice, a line in robots.txt that says no to GPTBot or ClaudeBot. That exists, but it's the smallest piece.
Where the 530 sites got stuck
| What happened | Sites | Share of all 3,693 |
|---|---|---|
| Bot check shown to our crawler | 216 | 5.8% |
| Homepage didn't load | 183 | 5.0% |
| robots.txt blocks some AI crawlers | 86 | 2.3% |
| robots.txt blocks all major AI crawlers | 29 | 0.8% |
| robots.txt couldn't be read | 16 | 0.4% |
Each site is counted once. The other 3,163 (85.6%) were open: robots.txt allowed AI crawlers, the homepage loaded, and nothing asked us to prove we were human.
Most of these owners didn't decide anything
A robots.txt rule is a decision someone made, even if it was a while ago. The other four rows usually aren't.
- Bot checks come from hosting and security settings. Cloudflare showed one on 112 sites and SiteGround on 99. Often the owner never turned them on themselves.
- Homepages that didn't load were mostly broken HTTPS: 140 of the 183 had a certificate or TLS error. A browser shows a warning page. A crawler just leaves.
- Unreadable robots.txt files returned an error or something that wasn't a robots.txt.
Put together, 415 of the 530 sites (78%) weren't fully open for reasons that have nothing to do with a choice about AI.
Why it matters
When someone asks ChatGPT or Gemini for "a dentist near me who takes my insurance," the model may look your site up live. If it hits a bot check or a certificate error, it answers from whatever else it can find: an old directory listing, a review site, a competitor's page. Your own words don't make it in.
What to check this week
- Open your site in a private browser window. If you see a certificate warning, call whoever hosts it. That fix is usually quick.
- Ask your host or web person whether a bot check or "bot fight" setting is on, and whether it lets known AI crawlers through.
- Look at yoursite.com/robots.txt. If you see "Disallow: /" under GPTBot, ClaudeBot or similar, decide whether you still want it.
- Run the free AI crawler check to see which group your site falls into.
Data: Preferly, "How many small business websites block AI crawlers?" (2026), https://www.preferly.ai/research/ai-crawler-study. We checked 4,111 small and local business websites in 26 US cities on Oct 22, 2026 and analyzed the 3,693 that responded. The aggregate data is free to reuse under CC BY 4.0 with that credit.
Questions
Does a bot check really stop AI tools?
It can. A bot check asks the visitor to prove it's a person. An AI tool fetching your page for an answer usually can't, so it gets the check page instead of your content.
How do I know which group my site is in?
Run the free AI crawler check. It reads your robots.txt and homepage the way an identified crawler does and tells you which of these results you get.
3 AI models, 5 customer questions, about a minute. Or check whether AI crawlers can reach your site.
No email. It reads your robots.txt and homepage, the way AI crawlers do.