AI robots.txt generator: choose which AI crawlers can read your site
Pick which AI crawlers may read your website and copy the robots.txt lines. Free, no email, nothing leaves your browser.
Your robots.txt lines
# AI crawlers let in User-agent: OAI-SearchBot Allow: / Disallow: /admin Disallow: /cart Disallow: /checkout Disallow: /account User-agent: ChatGPT-User Allow: / Disallow: /admin Disallow: /cart Disallow: /checkout Disallow: /account User-agent: Claude-User Allow: / Disallow: /admin Disallow: /cart Disallow: /checkout Disallow: /account User-agent: Claude-SearchBot Allow: / Disallow: /admin Disallow: /cart Disallow: /checkout Disallow: /account User-agent: PerplexityBot Allow: / Disallow: /admin Disallow: /cart Disallow: /checkout Disallow: /account User-agent: Perplexity-User Allow: / Disallow: /admin Disallow: /cart Disallow: /checkout Disallow: /account User-agent: Meta-ExternalFetcher Allow: / Disallow: /admin Disallow: /cart Disallow: /checkout Disallow: /account User-agent: Meta-WebIndexer Allow: / Disallow: /admin Disallow: /cart Disallow: /checkout Disallow: /account # AI crawlers kept out User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: /
- robots.txt lives at the root of your site, as
yoursite.com/robots.txt. Your website builder or host has a setting for it. - Add these lines to your file; don’t delete the rules already there. If an existing line names one of these crawlers, keep only one group per crawler.
- robots.txt is a request. Firewalls and CDN bot settings can still block crawlers it lets in, and crawlers that fetch a page for someone may not follow it. Check what AI crawlers see.
This tool answers one narrow question. To see what AI actually says about your business, and where AI contradicts your website, run the free check.
See what AI says about youThe full report checks 10 leading AI models and gives you every fix with paste-ready code, $99 once. See pricing
Other free tools: Business schema checker · Where to get listed
The short answer
Add a group to robots.txt for each AI crawler, with Allow: / to let it in or Disallow: / to keep it out. Search and assistant crawlers fetch pages for live answers; training crawlers collect pages for future models. Keeping training crawlers out doesn’t remove you from AI search.
What this tool doesn’t cover
What AI actually says about your business, and where AI contradicts your website. The free check asks AI models the questions your customers ask and shows both.
Questions
Which AI crawlers should a business let in?
If you want AI to recommend you, let in the search and assistant crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher. They fetch pages for live answers. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent) are a separate choice.
Will blocking GPTBot keep me out of ChatGPT answers?
No. OpenAI says GPTBot collects pages that may be used to train its models. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches a page when someone asks. Each one has its own robots.txt setting.
What is Google-Extended?
A robots.txt switch, not a separate crawler. It tells Google whether pages Googlebot already collects may be used for its Gemini models. Google says it doesn’t affect whether your site appears in Google Search.
Where does robots.txt go?
At the root of your website, as yoursite.com/robots.txt. Most website builders have a setting for it; on a custom site, whoever maintains it can add the lines.
How do I check it worked?
Run the free AI crawler check on your site. It reads your robots.txt and shows which AI crawlers it lets in, with the rule that decides each one.