Skip to content
Barkhausen AI
A vibrating-sample magnetometer with electromagnet pole pieces and pickup coils.

Reference

Reference

The AI crawlers behind answer engines, the official documentation that governs them, per-engine profiles, the vocabulary of the field, and the annotated primary literature — drawn from primary sources.

Jérôme · CC BY-SA 3.0

AI crawler registry

The AI-related crawlers by operator and function — training, retrieval, user-triggered fetch, agent, and search — with their robots.txt tokens and official documentation.

Last reviewed 2026

Engine documentation index

An annotated directory of the official documentation for AI answer engines and their crawlers.

Last reviewed 2026

Glossary

Canonical definitions of the field's terms, drawn from the published conventions.

Last reviewed 2026

Engine profiles

Per-engine reference profiles — the crawlers each operator publishes, and what its documentation states about retrieving web content and grounding answers in sources.

Last reviewed 2026

Annotated bibliography

The field's primary literature — measurement studies, statistical foundations, corpus and crawling research, adoption evidence, and the reporting-guideline tradition — each with a short appraisal and its source.

Last reviewed 2026