AI Crawler & Bot Directory
geoprompttracker.com- Category
- Education
- Rank
- No. 2309Tools index
- Type
- TOOL
- Date
About
This is a maintained, monthly-refreshed directory profiling 28 AI web crawlers and user agents, including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, from companies like OpenAI, Anthropic, Google, Meta, and ByteDance. Each entry explains what the bot does, whether it respects robots.txt, and provides copy-paste rules to allow or block it, alongside a checker for a site's current robots.txt.
Why it made the leaderboard
Gives site operators copy-paste robots.txt rules and compliance status for every major AI crawler in one place, useful for deciding what to allow or block.
Tags
ai-crawlersrobots-txtweb-scrapingbot-directory
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.