How do I let AI crawlers like ChatGPT access my site?
Make sure your robots.txt doesn’t disallow the AI crawlers you want, and that no bot check turns them away. With no rules for them, they’re allowed. OpenAI says sites that block OAI-SearchBot won’t appear in ChatGPT search answers, while blocking GPTBot only opts out of training. Claude and Perplexity use separate names the same way.
Updated October 22, 2026
The names, by company and job
These are the 14 names our free crawler check reads in your robots.txt.
| Company | Name | What it’s for | What blocking it does |
|---|---|---|---|
| OpenAI | GPTBot | Training | Opts you out of OpenAI’s model training. ChatGPT search isn’t affected. |
| OpenAI | OAI-SearchBot | Search | OpenAI says your site won’t be shown in ChatGPT search answers. |
| OpenAI | ChatGPT-User | A visit a user asked for | OpenAI says robots.txt rules may not apply, because a person asked. |
| Anthropic | ClaudeBot | Training | Opts you out of Anthropic’s model training. |
| Anthropic | Claude-User | A visit a user asked for | Claude can’t fetch your pages when someone asks it to. |
| Anthropic | Claude-SearchBot | Search | Claude’s search answers may use your site less, or less accurately. |
| Perplexity | PerplexityBot | Search | Perplexity won’t surface and link your pages in its search results. |
| Perplexity | Perplexity-User | A visit a user asked for | Perplexity says it generally ignores robots.txt, because a person asked. |
| Google-Extended | Gemini training and grounding | Opts out of Gemini training and grounding. Google Search isn’t affected. | |
| Apple | Applebot-Extended | Training | Opts you out of Apple’s AI training. |
| Meta | Meta-ExternalAgent | Training and indexing | Keeps Meta’s AI crawler out. |
| Meta | Meta-ExternalFetcher | A visit a user asked for | Meta AI can’t fetch your page when someone asks it to. |
| Meta | Meta-WebIndexer | Search | Meta says allowing it helps Meta AI cite and link to your content in its answers. |
| Hark | HarkBot | Crawler for Hark’s AI agent | Keeps Hark’s crawler out. |
Where robots.txt lives
At the root of your site: yoursite.com/robots.txt. Open it in a browser to see yours. How to edit it:
- WordPress: your SEO plugin’s robots.txt editor, or the file in your site’s root folder.
- Squarespace: it writes the file for you; check its crawler settings for an option that blocks AI crawlers.
- Wix: the robots.txt editor in its SEO settings.
- Shopify: the
robots.txt.liquidtemplate in your theme. - Webflow: the robots.txt field in your site’s SEO settings.
How rules are read
- A crawler follows the group with its own name, if there is one, and otherwise
User-agent: *. - Inside that group, the longest matching path wins.
- With no file at all (a 404), everything is allowed.
The lines to add
You don’t need any, but if you want to be explicit, this group lets the answer crawlers in:
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /To opt out of training as well, see Should I block AI crawlers like GPTBot?
Common mistakes
Disallow: /underUser-agent: *, left over from a staging site.- A security plugin or CDN setting that added AI blocks you didn’t write.
- Blocking the search crawlers while meaning to block training only.
User-requested visits
ChatGPT-User and Perplexity-User fetch a page because a person asked for it. OpenAI says robots.txt rules may not apply to ChatGPT-User. Changes also take time: OpenAI says about 24 hours for search.
robots.txt isn’t the only gate
A bot check (Cloudflare, SiteGround, Sucuri, Imperva and others) can turn AI away even when robots.txt lets it in. Our crawler check fetches your homepage as PreferlyBot and reports any bot check. See Is Cloudflare blocking AI crawlers from my website?
Questions
If I don’t have a robots.txt, can AI crawlers read my site?
Yes. With no robots.txt, crawlers that follow the standard treat everything as allowed.
Does “User-agent: *” cover AI crawlers?
Yes, unless a group names the crawler. A crawler follows the most specific group that names it.
Do AI crawlers obey robots.txt?
The major AI companies say their crawlers follow it. It’s a request, not a lock, so a bot that ignores it isn’t stopped by it.
Do I need to list every AI crawler?
No. Only add groups if you want different rules for some of them.
Sources
No email. It reads your robots.txt and homepage, the way AI crawlers do.