Reference
An annotated directory of the official documentation for AI answer engines and their crawlers.
Last reviewed 2026.
OpenAI
Documents OpenAI's four crawlers and their robots.txt tokens: GPTBot (training), OAI-SearchBot (surfacing sites in ChatGPT search), ChatGPT-User (real-time fetch for user actions), and OAI-AdsBot (validating ad landing pages).
Documents Googlebot and Google-Extended; Google-Extended governs use of content for AI training separately from search indexing. Now hosted under the developers.google.com/crawling hub.
The consolidated hub for Google's crawler and fetcher documentation, including user-triggered fetchers; the older /search/docs crawler pages now redirect here.
A dated log of changes to Google's crawlers and fetchers — the canonical source for tracking new or reclassified Google user-agents.
Google's stated guidance on how content appears in AI Overviews and AI Mode; eligibility follows ordinary Search indexing, with no additional technical requirements.
Anthropic
Documents ClaudeBot (training), Claude-SearchBot (improving search result quality), and Claude-User (fetching pages for user requests), and how site owners control each.
Perplexity
Documents PerplexityBot (surfaces and links sites in Perplexity search results; stated not to crawl for foundation models) and Perplexity-User (visits a page to answer a user's question and links it).
Microsoft
Bingbot underpins Copilot's web results.
Apple
Documents Applebot (powers Spotlight, Siri, and Safari search) and Applebot-Extended (controls whether crawled content is used to train Apple's generative models).
Amazon
Documents Amazonbot, which crawls to improve Amazon products and services and may be used to train Amazon AI models; it honors robots.txt opt-out directives.
Meta
Documents Meta's crawlers, including meta-externalagent (training) and meta-externalfetcher (user-triggered fetch), with their robots.txt tokens.
Mistral AI
Documents MistralAI-Index (crawls for indexing only, to power Mistral search) and MistralAI-User (fetches a page in real time to answer a user's question in Vibe).
DuckDuckGo
Documents DuckAssistBot, which crawls pages in real time for DuckDuckGo's AI-assisted answers (which cite their sources); the data is stated not to be used to train AI models.
Common Crawl
The crawler behind the Common Crawl corpus — the shared upstream of most open training datasets.