Free AI crawler checker Free, no sign-up
Can ChatGPT, Claude and Perplexity see your site?
Check 13 AI and search crawlers against your robots.txt, then copy the rules that let the right ones in.
Search crawlers vs training crawlers
AI companies run different crawlers for different jobs. Search crawlers (like OAI-SearchBot, PerplexityBot and Claude-SearchBot) decide whether you can be found and cited in AI answers. Training crawlers (like GPTBot, ClaudeBot and CCBot) collect content to train models. Blocking a training crawler doesn't remove you from AI search, and blocking a search crawler can.
Many sites block AI crawlers by accident through a security plugin, a CDN bot setting or an old robots.txt. This checker reads your robots.txt the way the crawlers do and shows each one as open or blocked.
Questions
Does robots.txt control everything?
No. Firewalls and CDN bot protection can also block crawlers even when robots.txt allows them. If you're allowed here but still invisible, check your CDN's bot settings.
Which crawlers matter most for being recommended?
Googlebot and Bingbot (Bing powers parts of ChatGPT and Copilot search), OAI-SearchBot for ChatGPT search, PerplexityBot and Claude-SearchBot.
Will allowing crawlers slow my site?
Rarely. Search crawlers fetch modest volumes. If load is a concern, rate-limit at the CDN rather than blocking outright.
More free tools
Summit Score
Grade how ready your website is for AI assistants and Google in 20 seconds. 20 checks, badges, and your fastest fixes.
Open the toolAI visibilityWhat does AI say about you?
Get 12 buyer questions to ask ChatGPT and Perplexity, then score how often you're named vs a competitor.
Open the toolAI implementationAI readiness quiz
Ten quick questions. Find out how ready your company is for AI and what to build first.
Open the tool