72 small business sites block GPTBot but let ChatGPT search in. How the split works
Short answer: 72 of the 3,370 small business sites with a readable robots.txt in our AI crawler study (2.1%) block GPTBot, OpenAI's training crawler, while still allowing OAI-SearchBot, the crawler ChatGPT search uses. That's the setup OpenAI documents for opting out of training without disappearing from ChatGPT search.
Two crawlers, two jobs
- GPTBot collects pages that may be used to train future models. Blocked by 105 sites (3.1%).
- OAI-SearchBot indexes pages so ChatGPT can show them in search answers. Blocked by 34 sites (1.0%).
- ChatGPT-User fetches a page when a user's question needs it right then. Blocked by 33 sites (1.0%).
So most sites that block GPTBot (72 of 105) leave the search crawler alone. Whether that was a deliberate choice or a default from a plugin or a host setting, the result is the same: they're out of training but still reachable for answers.
The same split for other AI companies
| Company | Training crawler (blocked by) | Search or user crawler (blocked by) |
|---|---|---|
| Anthropic | ClaudeBot (96, 2.8%) | Claude-SearchBot (32, 0.9%), Claude-User (43, 1.3%) |
| Perplexity | none separate | PerplexityBot (42, 1.2%), Perplexity-User (33, 1.0%) |
| Google-Extended (79, 2.3%) | Googlebot (not affected by Google-Extended) |
How to set it up
In robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Then check it with the free AI crawler check. Our guide to allowing AI crawlers in robots.txt has the full list of names.
Being readable is the first step. To see whether AI actually names your business, who it names instead and where it contradicts your website, get the full $99 AI visibility audit, paid once.
Data: Preferly, "How many small business websites block AI crawlers?" (2026), https://www.preferly.ai/research/ai-crawler-study. We checked 4,111 small and local business websites in 26 US cities on Oct 22, 2026 and analyzed the 3,693 that responded. The aggregate data is free to reuse under CC BY 4.0 with that credit.
Questions
If I block GPTBot, can ChatGPT still recommend me?
It can still find you through search if OAI-SearchBot is allowed. OpenAI documents the two separately for that reason.
Does the same split work for Claude and Gemini?
Similar splits exist. Anthropic lists ClaudeBot (training) separately from Claude-SearchBot and Claude-User. Google-Extended controls Gemini training without affecting Google Search.
3 AI models, 5 customer questions, about a minute. Or check whether AI crawlers can reach your site.
No email. It reads your robots.txt and homepage, the way AI crawlers do.