Glossary · web

retrieval crawler

Also: answer-time fetch, AI search crawler

A bot that fetches a page at the moment someone asks a question, so an assistant can cite it — distinct from a crawler collecting training data.

The two use different user-agents and the near-miss names are the trap: Applebot is the crawler while Applebot-Extended is the training opt-out, and the same split exists for Claude, Perplexity, and DuckDuckGo. Blocking a training token does not block retrieval, and blocking a whole family removes you from answers entirely.

The distinction lets a site take a coherent position: decline to be training data, accept being cited. Getting it wrong in robots.txt is silent — nothing errors, you simply stop appearing in answers and have no way to notice.