# 72 small business sites block GPTBot but let ChatGPT search in. How the split works

> 2.1% of small business sites we checked block OpenAI's training crawler while allowing its search crawler. What each one does and how to set the same split.

Canonical: https://www.preferly.ai/blog/gptbot-vs-oai-searchbot
Published: 2026-10-31
Last updated: 2026-10-31

By the Preferly team

**Short answer:** 72 of the 3,370 small business sites with a readable robots.txt in [our AI crawler study](https://www.preferly.ai/research/ai-crawler-study) (2.1%) block GPTBot, OpenAI's training crawler, while still allowing OAI-SearchBot, the crawler ChatGPT search uses. That's the setup OpenAI documents for opting out of training without disappearing from ChatGPT search.

## Two crawlers, two jobs

- **GPTBot** collects pages that may be used to train future models. Blocked by 105 sites (3.1%).
- **OAI-SearchBot** indexes pages so ChatGPT can show them in search answers. Blocked by 34 sites (1.0%).
- **ChatGPT-User** fetches a page when a user's question needs it right then. Blocked by 33 sites (1.0%).

So most sites that block GPTBot (72 of 105) leave the search crawler alone. Whether that was a deliberate choice or a default from a plugin or a host setting, the result is the same: they're out of training but still reachable for answers.

## The same split for other AI companies

| Company | Training crawler (blocked by) | Search or user crawler (blocked by) |
|---|---|---|
| Anthropic | ClaudeBot (96, 2.8%) | Claude-SearchBot (32, 0.9%), Claude-User (43, 1.3%) |
| Perplexity | none separate | PerplexityBot (42, 1.2%), Perplexity-User (33, 1.0%) |
| Google | Google-Extended (79, 2.3%) | Googlebot (not affected by Google-Extended) |

## How to set it up

In robots.txt:

```
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
```

Then check it with the [free AI crawler check](https://www.preferly.ai/ai-crawler-check). Our guide to [allowing AI crawlers in robots.txt](https://www.preferly.ai/allow-ai-crawlers-in-robots-txt) has the full list of names.

Being readable is the first step. To see whether AI actually names your business, who it names instead and where it contradicts your website, get the [full $99 AI visibility audit](https://www.preferly.ai/ai-visibility-audit), paid once.

_Data: Preferly, "How many small business websites block AI crawlers?" (2026), https://www.preferly.ai/research/ai-crawler-study. We checked 4,111 small and local business websites in 26 US cities on Oct 22, 2026 and analyzed the 3,693 that responded. The aggregate data is free to reuse under CC BY 4.0 with that credit._

## Questions

### If I block GPTBot, can ChatGPT still recommend me?

It can still find you through search if OAI-SearchBot is allowed. OpenAI documents the two separately for that reason.

### Does the same split work for Claude and Gemini?

Similar splits exist. Anthropic lists ClaudeBot (training) separately from Claude-SearchBot and Claude-User. Google-Extended controls Gemini training without affecting Google Search.