Reference
AI crawler registry
The AI-related crawlers by operator and function — training, retrieval, user-triggered fetch, agent, and search — with their robots.txt tokens and official documentation.
Last reviewed 2026. Facts are from vendor documentation and public observation.
AI-related crawlers do different jobs. Training crawlers collect content to train models; retrieval crawlers build an index that AI answers draw from; user-fetch crawlers retrieve a page in real time to answer a specific prompt. Blocking one class does not block the others — see the note on robots.txt and AI crawlers.
Training
| Crawler | Operator | robots token | Documentation |
|---|---|---|---|
| GPTBot | OpenAI | GPTBot | official |
| ClaudeBot | Anthropic | ClaudeBot | official |
| Google-Extended | Google-Extended | official | |
| CCBot | Common Crawl | CCBot | official |
| Applebot-Extended | Apple | Applebot-Extended | official |
| Amazonbot | Amazon | Amazonbot | official |
| meta-externalagent | Meta | meta-externalagent | official |
| AI2Bot | Allen Institute for AI | AI2Bot | official |
| Bytespider | ByteDance | Bytespider | — |
| xAI / Grok | xAI | — | — |
| DeepSeek | DeepSeek | — | — |
| Moonshot (Kimi) | Moonshot AI | — | — |
| Qwen | Alibaba | — | — |
| cohere-ai | Cohere | cohere-ai | — |
Retrieval / search index
| Crawler | Operator | robots token | Documentation |
|---|---|---|---|
| OAI-SearchBot | OpenAI | OAI-SearchBot | official |
| Claude-SearchBot | Anthropic | Claude-SearchBot | official |
| PerplexityBot | Perplexity | PerplexityBot | official |
| Google-CloudVertexBot | Google-CloudVertexBot | official | |
| DuckAssistBot | DuckDuckGo | DuckAssistBot | official |
| YouBot | You.com | YouBot | official |
| MistralAI-Index | Mistral AI | MistralAI-Index | official |
| Diffbot | Diffbot | Diffbot | official |
| Timpibot | Timpi | Timpibot | — |
| omgili | Webz.io | omgili | — |
User-triggered fetch
| Crawler | Operator | robots token | Documentation |
|---|---|---|---|
| ChatGPT-User | OpenAI | ChatGPT-User | official |
| OAI-AdsBot | OpenAI | OAI-AdsBot | official |
| Claude-User | Anthropic | Claude-User | official |
| Perplexity-User | Perplexity | Perplexity-User | official |
| Google-NotebookLM | — | official | |
| meta-externalfetcher | Meta | meta-externalfetcher | official |
| MistralAI-User | Mistral AI | MistralAI-User | official |
Search index
Agent
| Crawler | Operator | robots token | Documentation |
|---|---|---|---|
| Google-Agent | — | official |
Agent traffic and undocumented crawlers
As of July 2026 only one agent-class crawler is officially documented and user-agent–identifiable, Google-Agent; other agent products present as ordinary browser traffic with no distinct crawler token (OpenAI's agent and Perplexity's Comet among them), so agent activity is largely not identifiable from user-agent strings alone.
Several AI operators publish no official crawler documentation: ByteDance's Bytespider carries a documented token but no operator documentation, and xAI, DeepSeek, Moonshot, and Alibaba document nothing. DeepSeek publishes no crawler user-agent at all and crawls anonymously. Rows without a verified user-agent token are recorded here from public observation and are omitted from any user-agent-based control the registry supports.