Cloudflare's new Bot Preference Sync keeps your robots.txt automatically aligned with whatever AI bot preferences you've already set in the dashboard — Search, Agent, and Training.
No more maintaining a static file. Set it once. We keep it in sync. 🔄
https://t.co/bo174jyzvJ
Crawler policy is now split by purpose, and the hand maintained file was the part most likely to be wrong.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
postCloudflare AI Search goes GA with image embeddings and PDF OCR
articleWe are introducing Clef and Clef-flash, open-source decision models hosted on…
articleToday, we’re making the Cloudflare Monetization Gateway available as part of a…
articleCloudflare Auto Router picks cheaper models per request in AI Gateway
