How many small business websites block AI crawlers?

We checked 4,111 websites of small and local businesses in 26 US cities, and could analyze 3,693 of them. 86% were fully open to AI crawlers. The other 14% weren’t: 5.8% showed our clearly identified crawler a bot check, 5.0% didn’t load at all (mostly because of broken HTTPS certificates), 3.1% block some or all AI crawlers in robots.txt, and 0.4% had a robots.txt we couldn’t read. Only the robots.txt blocks are a rule written for crawlers. The bot checks come from hosting and security vendors, and the broken homepages are certificate and server errors, so most of those owners never decided anything about AI. Their sites just aren’t fully reachable for it.

1. Deliberate blocking is rare. 3.2% of sites with a readable robots.txt block at least one of the 9 major AI crawlers from their homepage. 0.9% block all of them. When sites do block, 83% use rules that name AI crawlers rather than a blanket rule for every bot. 24% of the sites that block any AI crawler carry a section written by Cloudflare’s managed robots.txt setting, which the owner may never have looked at.

2. Training crawlers get blocked more than search crawlers. GPTBot, OpenAI’s training crawler, is the most blocked major crawler (3.1%), ahead of ClaudeBot (2.8%) and Google-Extended (2.3%). The crawlers that fetch pages for live answers are blocked about a third as often: OAI-SearchBot 1.0%, ChatGPT-User 1.0%, Claude-SearchBot 0.9%, PerplexityBot 1.2%. 2.1% of sites block training crawlers while leaving the search and user crawlers allowed, the split OpenAI documents for opting out of training while staying in ChatGPT search.

3. Bot checks turn away more AI visitors than robots.txt does. 5.8% of sites showed our crawler a bot check instead of their homepage: Cloudflare on 3.0% of sites and SiteGround on 2.7%, with Akamai and AWS WAF on a handful. 40% of the sites we analyzed run behind Cloudflare. A bot check doesn’t always stop the big AI crawlers, which vendors can recognize and let through. It does stop newer AI tools, and it will stop AI agents acting for a customer unless the vendor lets them in.

4. One in 20 homepages didn’t load. 5.0% of sites failed to load for an automated visitor over HTTPS. Most of those (3.8% of all sites analyzed) had an HTTPS certificate problem: a certificate for a different name, an expired one, or a failed secure connection. The rest returned server errors or redirected in a loop. A browser shows a person a warning page for these. An AI crawler usually just moves on.

5. Missing robots.txt isn’t a problem. 12% of sites have no robots.txt. That’s fine: under the robots.txt standard (RFC 9309), no file means every crawler is allowed.

6. Restaurants, hotels and law firms see the most bot checks. 9.6% of restaurant and bar sites, 9.1% of law firm sites and 8.9% of hotel sites showed a bot check, against 1.9% of vet and pet sites. Real estate and hotels had the most homepages that didn’t load (9.9% and 9.4%). Cafes and bakeries, and vets and pet services, were the most open, with 90% and 91% of sites fully reachable.

Preferly, October 2026. 3,693 US small and local business sites analyzed. Sample drawn from OpenStreetMap data, © OpenStreetMap contributors, ODbL.

• Check what AI crawlers get, not just your robots.txt. Most sites that aren’t fully open to AI crawlers got that way through a bot check or a broken certificate, not a rule the owner wrote.
• If you use Cloudflare, SiteGround or another firewall, look at its bot settings. Allow the AI search and user crawlers you want to be found by (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User). You can still block training crawlers like GPTBot.
• Fix certificate errors first. A site that doesn’t load over HTTPS loses AI visitors and people alike.
• You don’t need a robots.txt to be found. If you have one, make sure it doesn’t disallow the crawlers you want.
• See what AI crawlers get from your own site with our free AI crawler check. It runs the same two requests we used in this study, and needs no email.

Sample. We took businesses with a website listed in OpenStreetMap, in 14 categories (restaurants and bars, cafes and bakeries, dentists, doctors and clinics, lawyers, accounting and insurance, real estate, home services and trades, auto repair, hair and beauty, fitness, retail shops, hotels, and vets and pet services), across 26 US cities and metro areas from Boston to San Jose and Miami to Seattle. We dropped anything tagged with a brand, any domain listed for 3 or more locations (chains and franchises), social media and booking pages, and government and school sites. We then drew up to 25 sites per city and category and 160 per city, with a fixed random seed, before checking any site. That gave 4,111 websites.

What we checked. For each site, on October 22, 2026, we made the same two requests Preferly’s free AI crawler check makes: robots.txt, then the homepage. Sites that showed a bot check, an error or a timeout were checked once more a few minutes later, and the second result is the one we report. We read robots.txt by the rules of RFC 9309 for 13 AI crawlers: the 9 major ones (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User and Google-Extended) and 4 others (Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher and HarkBot). A crawler counts as blocked when the rules disallow it from the homepage. We fetched the homepage only when robots.txt allowed our own crawler. We recognized bot checks by the vendor’s own markers in the response.

How we identified ourselves. Every request said it came from PreferlyBot, with a link to a page explaining what it does (preferly.ai/bot). We didn’t pretend to be a browser or another company’s crawler, and we didn’t try to get past any bot check. Requests weren’t signed with Web Bot Auth. Requests were spread out, with no more than one at a time per site. We kept no personal data, and we don’t name any business.

How to read the percentages. Percentages about robots.txt rules are out of the 3,370 analyzed sites whose robots.txt we could read (or that had none). Everything else is out of the 3,693 sites analyzed, so the chart of what crawlers find shows slightly lower blocking shares.

What we left out. 246 listed websites (6.0%) were on domains that no longer exist. Another 172 didn’t respond in a way we could interpret from our network: some timed out or reset the connection, and some redirected to plain HTTP, which our network couldn’t follow. We left both groups out, leaving 3,693 sites analyzed.

• OpenStreetMap lists the businesses volunteers have mapped, and only those with a website. That leans toward city centers and businesses that have a site at all.
• A bot check shown to our crawler may not be shown to GPTBot or Googlebot, which firewall vendors can recognize by their published addresses. Our bot-check numbers show what a new, honestly identified automated visitor gets.
• Firewalls can also block a specific AI crawler at the network edge without any sign in robots.txt or in what our crawler saw. We couldn’t measure that without pretending to be that crawler, which we don’t do. So the true share of sites that aren’t fully open to AI crawlers is, if anything, higher.
• HTTPS only: sites that work only over plain HTTP were left out as inconclusive.
• Each site was checked once, on October 22, 2026. Sites change.

The totals by category and verdict, without any site names or domains:

Download the data (CSV, CC BY 4.0)

The aggregate data is free to reuse under the Creative Commons Attribution 4.0 International license (CC BY 4.0). Please credit: Preferly, "How many small business websites block AI crawlers?" (2026), https://www.preferly.ai/research/ai-crawler-study. Sample drawn from OpenStreetMap data, © OpenStreetMap contributors, ODbL.