# How do I let AI crawlers like ChatGPT access my site?

> The robots.txt names that decide whether ChatGPT, Claude, Perplexity and Gemini can read your site, how rules are read, and what else can block AI.

Canonical: https://www.preferly.ai/allow-ai-crawlers-in-robots-txt
Last updated: 2026-10-22

Make sure your robots.txt doesn’t disallow the AI crawlers you want, and that no bot check turns them away. With no rules for them, they’re allowed. OpenAI says sites that block OAI-SearchBot won’t appear in ChatGPT search answers, while blocking GPTBot only opts out of training. Claude and Perplexity use separate names the same way.

Updated October 22, 2026

## The names, by company and job

These are the 14 names our free crawler check reads in your robots.txt.

| Company | Name | What it’s for | What blocking it does |
| --- | --- | --- | --- |
| OpenAI | GPTBot | Training | Opts you out of OpenAI’s model training. ChatGPT search isn’t affected. |
| OpenAI | OAI-SearchBot | Search | OpenAI says your site won’t be shown in ChatGPT search answers. |
| OpenAI | ChatGPT-User | A visit a user asked for | OpenAI says robots.txt rules may not apply, because a person asked. |
| Anthropic | ClaudeBot | Training | Opts you out of Anthropic’s model training. |
| Anthropic | Claude-User | A visit a user asked for | Claude can’t fetch your pages when someone asks it to. |
| Anthropic | Claude-SearchBot | Search | Claude’s search answers may use your site less, or less accurately. |
| Perplexity | PerplexityBot | Search | Perplexity won’t surface and link your pages in its search results. |
| Perplexity | Perplexity-User | A visit a user asked for | Perplexity says it generally ignores robots.txt, because a person asked. |
| Google | Google-Extended | Gemini training and grounding | Opts out of Gemini training and grounding. Google Search isn’t affected. |
| Apple | Applebot-Extended | Training | Opts you out of Apple’s AI training. |
| Meta | Meta-ExternalAgent | Training and indexing | Keeps Meta’s AI crawler out. |
| Meta | Meta-ExternalFetcher | A visit a user asked for | Meta AI can’t fetch your page when someone asks it to. |
| Meta | Meta-WebIndexer | Search | Meta says allowing it helps Meta AI cite and link to your content in its answers. |
| Hark | HarkBot | Crawler for Hark’s AI agent | Keeps Hark’s crawler out. |

## Where robots.txt lives

At the root of your site: `yoursite.com/robots.txt`. Open it in a browser to see yours. How to edit it:

- WordPress: your SEO plugin’s robots.txt editor, or the file in your site’s root folder.
- Squarespace: it writes the file for you; check its crawler settings for an option that blocks AI crawlers.
- Wix: the robots.txt editor in its SEO settings.
- Shopify: the `robots.txt.liquid` template in your theme.
- Webflow: the robots.txt field in your site’s SEO settings.

## How rules are read

- A crawler follows the group with its own name, if there is one, and otherwise `User-agent: *`.
- Inside that group, the longest matching path wins.
- With no file at all (a 404), everything is allowed.

## The lines to add

You don’t need any, but if you want to be explicit, this group lets the answer crawlers in:

```
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
```

To opt out of training as well, see Should I block AI crawlers like GPTBot?

To write the lines for your own choice of crawlers, use the free AI robots.txt generator.

## Common mistakes

- `Disallow: /` under `User-agent: *`, left over from a staging site.
- A security plugin or CDN setting that added AI blocks you didn’t write.
- Blocking the search crawlers while meaning to block training only.

## User-requested visits

ChatGPT-User and Perplexity-User fetch a page because a person asked for it. OpenAI says robots.txt rules may not apply to ChatGPT-User. Changes also take time: OpenAI says about 24 hours for search.

## robots.txt isn’t the only gate

A bot check (Cloudflare, SiteGround, Sucuri, Imperva and others) can turn AI away even when robots.txt lets it in. In our study of 3,693 small and local business websites, 5.8% showed our identified crawler a bot check. Our crawler check fetches your homepage as PreferlyBot and reports any bot check. See Is Cloudflare blocking AI crawlers from my website?

## Questions

### If I don’t have a robots.txt, can AI crawlers read my site?

Yes. With no robots.txt, crawlers that follow the standard treat everything as allowed.

### Does “User-agent: *” cover AI crawlers?

Yes, unless a group names the crawler. A crawler follows the most specific group that names it.

### Do AI crawlers obey robots.txt?

The major AI companies say their crawlers follow it. It’s a request, not a lock, so a bot that ignores it isn’t stopped by it.

### Do I need to list every AI crawler?

No. Only add groups if you want different rules for some of them.

## Sources

- [RFC 9309, “Robots Exclusion Protocol”](https://www.rfc-editor.org/rfc/rfc9309)
- [OpenAI, “Overview of OpenAI Crawlers”](https://developers.openai.com/api/docs/bots)
- [Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity, “Perplexity Crawlers”](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [Google, “Google’s common crawlers”](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Meta for Developers, “Meta Web Crawlers”](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers)

## Related

- Should I block AI crawlers like GPTBot from my website?
- Is Cloudflare blocking AI crawlers from my website?
- What is an llms.txt file, and do I need one?
- How do I get my business into Google AI Overviews, and fix what they say?
- How do I know if AI web crawlers are blocked from accessing my site and spreading false information about my products?
- All guides
- Pricing: the free check and the full report