# AI robots.txt generator: choose which AI crawlers can read your site

> Pick which AI crawlers may read your website and copy the robots.txt lines: let AI search in and keep training out, or choose each one. Free, no email.

Canonical: https://www.preferly.ai/tools/ai-robots-txt-generator

Pick which AI crawlers may read your website and copy the robots.txt lines. Free, no email, nothing leaves your browser.

## The short answer

Add a group to robots.txt for each AI crawler, with Allow: / to let it in or Disallow: / to keep it out. Search and assistant crawlers fetch pages for live answers; training crawlers collect pages for future models. Keeping training crawlers out doesn’t remove you from AI search.

## What this tool doesn’t cover

What AI actually says about your business, and where AI contradicts your website. The free check asks AI models the questions your customers ask and shows both.

## Questions

### Which AI crawlers should a business let in?

If you want AI to recommend you, let in the search and assistant crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher. They fetch pages for live answers. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent) are a separate choice.

### Will blocking GPTBot keep me out of ChatGPT answers?

No. OpenAI says GPTBot collects pages that may be used to train its models. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches a page when someone asks. Each one has its own robots.txt setting.

### What is Google-Extended?

A robots.txt switch, not a separate crawler. It tells Google whether pages Googlebot already collects may be used for its Gemini models. Google says it doesn’t affect whether your site appears in Google Search.

### Where does robots.txt go?

At the root of your website, as yoursite.com/robots.txt. Most website builders have a setting for it; on a custom site, whoever maintains it can add the lines.

### How do I check it worked?

Run the free AI crawler check on your site. It reads your robots.txt and shows which AI crawlers it lets in, with the rule that decides each one.

## Related

- How do I know if AI web crawlers are blocked from accessing my site and spreading false information about my products?
- Should I block AI crawlers like GPTBot from my website?
- How do I let AI crawlers like ChatGPT access my site?
- All guides
- Pricing: the free check and the full report