Reference
Engine profiles
Per-engine reference profiles — the crawlers each operator publishes, and what its documentation states about retrieving web content and grounding answers in sources.
Last reviewed 2026.
Each profile records what the operator states in its own documentation — the crawlers it publishes, how it says it retrieves web content, and what it discloses about grounding answers in sources. These are documentation summaries, not measurements of behavior.
OpenAI
ChatGPT
- Retrieval behavior
- OpenAI documents OAI-SearchBot as the crawler that surfaces websites in ChatGPT's search features, and ChatGPT-User as fetching a page in real time when a user action requires it rather than crawling automatically. GPTBot is documented separately as a training crawler.
- Grounding disclosed
- OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that ChatGPT-User is not used to determine whether content may appear in search — the two roles are documented as distinct.
Documentation
Google
Google AI features (AI Overviews, AI Mode) / Gemini
- Retrieval behavior
- Google documents that AI features in Search draw on the Google Search index: to be eligible for AI Overviews or AI Mode a page must be indexed and eligible to be shown in Search with a snippet, with no additional technical requirements. Google-Extended governs whether crawled content is used for Gemini and Vertex AI training separately from Search indexing.
- Grounding disclosed
- Google documents that AI Overviews and AI Mode surface links to help people find sources and explore content, and that eligibility follows ordinary Search indexing rather than a separate opt-in.
Documentation
Microsoft
Microsoft Copilot
- Retrieval behavior
- Microsoft documents Bingbot as the crawler behind the Bing search index; Copilot's web results are drawn from Bing. Microsoft's crawler documentation does not describe a distinct Copilot crawler user-agent.
- Grounding disclosed
- Grounding of Copilot's answers is not described on the Bing crawler documentation; what is documented is that Bingbot underpins the Bing web index that Copilot draws on.
Documentation
Perplexity
Perplexity
- Retrieval behavior
- Perplexity documents PerplexityBot as designed to surface and link websites in Perplexity search results, and states it is not used to crawl content for AI foundation models. Perplexity-User visits a web page when a user's question requires it.
- Grounding disclosed
- Perplexity states that Perplexity-User may visit a page to help provide an accurate answer and includes a link to that page in its response, and that allowing PerplexityBot is what makes a site eligible to appear in results.
Documentation
Anthropic
Claude
- Retrieval behavior
- Anthropic documents Claude-SearchBot as navigating the web to improve search result quality for users, and Claude-User as accessing websites when individuals ask Claude questions. ClaudeBot is documented separately as a training crawler.
- Grounding disclosed
- Anthropic documents each crawler's purpose and how site owners control it: Claude-User is described as accessing sites on behalf of a user's question, and Claude-SearchBot as analyzing content to enhance the relevance and accuracy of search responses.
Documentation
DuckDuckGo
DuckDuckGo (Duck.ai / DuckAssist)
- Retrieval behavior
- DuckDuckGo documents DuckAssistBot as a web crawler for DuckDuckGo Search that crawls pages in real time for its AI-assisted answers.
- Grounding disclosed
- DuckDuckGo states that its AI-assisted answers prominently cite their sources, and that the crawled data is not used in any way to train AI models.
Documentation