AI crawler
Also called AI bot, AI user agent, LLM crawler, GPTBot, ClaudeBot, PerplexityBot.
DefinitionAn AI crawler is an automated bot run by an AI company that fetches web pages to train models, build an AI search index or answer a user's request.
An AI crawler is an automated bot run by an AI company that fetches web pages, either to collect training data for models, to build an index for AI search, or to load a page a user asked about. Each crawler identifies itself with a user agent name, and site owners can allow or block them in robots.txt. Blocking the wrong one can quietly remove a site from AI answers.
Which AI crawlers should site owners know?
OpenAI
- GPTBot collects content that may be used to train OpenAI’s models.
- OAI-SearchBot finds and indexes pages to show and link in ChatGPT’s search features. It is not used for training.
- ChatGPT-User fetches a page when a ChatGPT user’s request needs it, rather than crawling on its own. Because these visits are user-initiated, check OpenAI’s current documentation for how robots.txt applies.
Anthropic
- ClaudeBot collects public web content that may be used to train Claude models.
- Claude-SearchBot crawls to improve the quality of search results in Claude.
- Anthropic also uses Claude-User for pages fetched in response to a user’s request.
Perplexity
- PerplexityBot indexes pages so they can be shown and linked in Perplexity’s answers. Perplexity says it is not used to train foundation models.
- Google-Extended is a robots.txt control token, not a separate crawler. Googlebot does the crawling. Disallowing Google-Extended tells Google not to use your content to train Gemini models or ground answers in Gemini apps. It does not remove your site from Google Search, and it does not remove you from AI Overviews or AI Mode, which are part of Search.
Should you block AI crawlers?
That is a business decision. Many brands block training crawlers (GPTBot, ClaudeBot) but allow search and user crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot), so they can still be cited in answers. The LLM SEO guide covers how crawl access fits into the wider work. The common mistake is a blanket rule, or a firewall or bot-protection setting, that blocks everything without anyone noticing.
Example
A hypothetical furniture retailer, Oakwell, has a robots.txt rule copied from its staging site that disallows all user agents from its /products/ folder, plus a bot-protection setting that challenges unknown agents. OAI-SearchBot and PerplexityBot can’t read any product page, and none are cited in answers. After both are fixed, the pages can be indexed again, and the agency tracks whether citations follow.
How AI Visibility Desk checks access
Every AI Visibility Desk audit includes an AI crawler access check, and the built-in site audit flags AI crawler blocking alongside other issues. The free SEO checker runs a quick version.
Read the full guide
Related terms
Measure it on a real brand
Run a free AI visibility audit across ChatGPT, Gemini, Google AI Overviews, AI Mode, Perplexity and Claude, and get a white-label report.
Start free