# 216 small business sites showed our crawler a bot check. Here's why that hides you from AI

> 5.8% of the 3,693 small business websites we checked showed an identified crawler a bot check instead of the homepage. Who causes it and what to ask your host.

Canonical: https://www.preferly.ai/blog/bot-checks-hide-small-business-sites-from-ai
Published: 2026-10-29
Last updated: 2026-10-29

By the Preferly team

**Short answer:** 216 of the 3,693 small business websites we checked (5.8%) showed our crawler a bot check instead of the homepage. That's more than the 115 sites (3.1%) that block AI crawlers on purpose in robots.txt. Two vendors account for almost all of them.

## Who showed the bot checks

| Vendor | Sites | Share of all 3,693 |
|---|---|---|
| Cloudflare | 112 | 3.0% |
| SiteGround | 99 | 2.7% |
| Akamai | 3 | 0.1% |
| AWS WAF | 2 | 0.1% |

Our crawler, PreferlyBot, says who it is in every request and follows robots.txt. It still got stopped, because a bot check doesn't read robots.txt. It looks at the visitor and decides whether it seems human.

## Why this is easy to miss

From your side, nothing looks wrong. You open your site in a browser, it loads, customers book. The check only shows up for automated visitors, and AI tools fetching a page to answer a question are automated visitors.

SiteGround's check is especially quiet. It sends back a tiny page with a success code that redirects to a captcha. To a simple tool that looks like "the page loaded," which is why a simple check can report these sites as fine.

## Which kinds of business hit it most

Restaurants and bars had the highest rate in our sample (33 of 342, 9.6%), followed by lawyers (22 of 243, 9.1%), hotels (18 of 202, 8.9%) and dentists (24 of 287, 8.4%). Vets and pet services had the lowest (4 of 207, 1.9%).

## What to ask your host or web person

- "Is there a bot check, captcha or 'bot fight mode' on our site?"
- "Does it let verified bots through, including AI crawlers like GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot?"
- "Can we allow those while keeping protection on for everything else?"

Then run the [free AI crawler check](https://www.preferly.ai/ai-crawler-check) again to confirm. If you use Cloudflare, our page on [whether Cloudflare is blocking AI crawlers](https://www.preferly.ai/is-cloudflare-blocking-ai-crawlers) walks through the settings.

_Data: Preferly, "How many small business websites block AI crawlers?" (2026), https://www.preferly.ai/research/ai-crawler-study. We checked 4,111 small and local business websites in 26 US cities on Oct 22, 2026 and analyzed the 3,693 that responded. The aggregate data is free to reuse under CC BY 4.0 with that credit._ A bot check is what a new, clearly identified crawler sees. Large AI crawlers may be treated differently by some vendors.

## Questions

### Is a bot check the same as blocking AI in robots.txt?

No. robots.txt is a public rule that well-behaved crawlers read and follow. A bot check is a page your host or security service shows instead of your content. It can stop crawlers you never meant to block.

### Should I turn bot protection off?

Not necessarily. It protects against real abuse. Ask your host whether it can let verified AI crawlers and search bots through while still stopping the rest.