Should I block AI crawlers like GPTBot from my website?
It depends on what you want to block. Blocking GPTBot opts you out of OpenAI’s training, not ChatGPT search. If you want customers to find you through AI, keep the search and user-requested crawlers open, such as OAI-SearchBot and ChatGPT-User. You can block training crawlers separately without leaving AI answers.
Updated October 22, 2026
Two different decisions
Some AI crawlers collect pages to train models. Others fetch pages to answer a question someone is asking right now, in search or because a person asked for that page. Each company uses a separate name for each job, so you can decide them separately.
| Company | Training | Answers (search and user-requested visits) |
|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot, Claude-User |
| Perplexity | None | PerplexityBot, Perplexity-User |
| Google-Extended (Gemini training and grounding; doesn’t affect Google Search) | Google Search uses Googlebot | |
| Apple | Applebot-Extended | |
| Meta | Meta-ExternalAgent |
When blocking training makes sense
When your content is the product, such as writing, courses or original photos, or when your industry’s rules restrict reuse.
When it hurts you
A local business whose website is mostly facts (services, hours, prices) gains little from blocking training, and can lose answers if the wrong names are blocked. The real risk is a broad rule or a bot check that blocks the answer crawlers too.
Check what’s blocking you today
Your CDN or a security plugin may have added rules you didn’t write. See Is Cloudflare blocking AI crawlers from my website?
How to do it
Paste this into your robots.txt to stay in AI answers and opt out of training:
# Allow AI search and user fetches; opt out of training
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /Only block what you mean to block. Changes take effect when each crawler next reads the file; OpenAI says about 24 hours for search.
Check yours
The free AI crawler check reads your robots.txt for 14 AI crawlers and loads your homepage the way they do. No email.
Questions
Does blocking GPTBot hurt my Google ranking?
No. GPTBot is OpenAI’s. Google Search uses Googlebot, and Google says Google-Extended doesn’t affect Search either.
Will blocking AI crawlers stop my content being used?
It asks compliant crawlers not to fetch it. It doesn’t remove what was collected before, and it doesn’t stop bots that ignore robots.txt.
Does ChatGPT-User follow robots.txt?
OpenAI says ChatGPT-User acts for a user, so robots.txt rules may not apply. A firewall or bot check can still block it.
What do most small businesses do?
In our study, 97% of small business sites with a readable robots.txt left every major AI crawler open. See the study.
Sources
No email. It reads your robots.txt and homepage, the way AI crawlers do.