Are AI crawlers blocked from your site?
Enter your website. We check your robots.txt rules for 14 AI crawlers and whether your homepage loads for an automated visitor. Free, no email, results in a few seconds.
How do I know if AI web crawlers are blocked from accessing my site and spreading false information about my products?
Check two things: your robots.txt, which tells each AI crawler whether it may read your pages, and whether your site shows automated visitors a bot challenge. The checker above does both. If AI crawlers can’t read your site, AI answers about your products come from other sources: old directory listings, review sites, articles. That’s where answers that contradict your website usually come from.
Blocked crawlers, step by step
- Robots.txt. Each AI company publishes the name its crawler uses (GPTBot, ClaudeBot, PerplexityBot and others). A
Disallow: /under that name, or underUser-agent: *, keeps it out. Should I block GPTBot? - Bot protection. Firewalls and CDN settings can block AI crawlers even when robots.txt allows them. Some CDNs turn this on by default.
- JavaScript-only content. Many AI tools read the raw page without running scripts, so facts that only appear after scripts run can be missed.
Checking whether AI contradicts your website
Unblocking crawlers lets AI read your site. To see whether AI is already telling customers something different, ask it about your business. Preferly’s free check asks 3 AI models and shows up to 2 places where AI contradicts your website; the full report checks every claim.