Can AI crawlers read your site?
One line in robots.txt can keep a site out of ChatGPT, Claude or Perplexity answers. Check which AI bots are allowed, in seconds.
AI crawlers in robots.txt
Site signals
Crawlers are step one. See whether AI actually names the brand, across six engines, with a full audit.
Run a free full auditThe AI crawlers that matter
Each company runs more than one bot. Training, search and user-triggered fetching are often separate, so you can allow one and block another.
| Feature | Run by | Used for |
|---|---|---|
| GPTBot | OpenAI | Collecting content for model training |
| OAI-SearchBot | OpenAI | Search results in ChatGPT |
| ChatGPT-User | OpenAI | Fetching a page a user asked about |
| ClaudeBot | Anthropic | Collecting content for model training |
| Claude-SearchBot | Anthropic | Search results in Claude |
| PerplexityBot | Perplexity | Indexing pages for Perplexity answers |
| Google-Extended | A robots.txt token for Gemini model use, not a separate crawler |
Read each company’s crawler documentation for the current list; they add and rename bots.
Why AI crawler access matters
AI engines that search the web can only cite pages their crawlers are allowed to fetch. If robots.txt blocks OAI-SearchBot, ChatGPT search can’t read the site; if it blocks PerplexityBot, Perplexity can’t index it. Many sites block these bots by accident, through a security plugin, a CDN setting or an old “block all bots except Google” rule.
How to allow AI crawlers in robots.txt
Remove any group that disallows the bot, or add an explicit allow group, for example User-agent: OAI-SearchBot followed by Allow: /. Then check your CDN or firewall: some block AI bots before robots.txt is even read. Run this checker again after the change.
What is llms.txt?
A plain-text file at the root of a site that summarises what the site is and links to its most useful pages, written for AI assistants. It is optional and engines treat it differently, but it is cheap to add. Use our llms.txt generator to draft one, and read the llms.txt definition or the AI crawler glossary entry for more.
Questions
How do I know if my site blocks GPTBot?
Enter your site above. The checker reads your robots.txt and reports, for GPTBot and the other main AI crawlers, whether each is allowed, blocked or partly blocked. You can also open yoursite.com/robots.txt and look for a “User-agent: GPTBot” group followed by “Disallow: /”.
Should I block AI crawlers?
It depends on what you want. Blocking a search crawler such as OAI-SearchBot or PerplexityBot can keep a site out of that engine’s answers. Blocking a training crawler such as GPTBot affects use in future model training. Most businesses that want to be recommended by AI allow the search crawlers at least.
What does “partly blocked” mean?
The crawler is allowed on some paths and disallowed on others, for example blocked from /blog/ but allowed elsewhere. Check that the paths you want cited are open.
What else does the checker look at?
Whether the site has an llms.txt file and a sitemap, and whether the homepage is reachable with a title, a meta description, structured data types and text content in the HTML itself rather than only after JavaScript runs.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended is a robots.txt control for use of content in Gemini models, not a separate crawler. AI Overviews and AI Mode use Google Search’s normal crawling, so blocking Google-Extended does not remove a site from them.
Run your first audit free
Enter a client’s website, let the AI fill in competitors and buyer questions, and get a branded report across six AI engines. No card needed.
Start free