Cloudflare's AI Search Initiatives: Smarter Crawling Signals and Pay Per Use
Source
Matthew Conroy
Author
Matthew Conroy
Date
Key takeaways · AI-distilled
Pew found that when Google shows an AI summary, users click a traditional result 8% of the time, roughly half the rate without a summary, and click a link inside the summary only 1%. The visit that funded the page mostly stops happening.
Crawling is a poor proxy for value. One page can be fetched once and cited in thousands of answers, or fetched repeatedly and never used. Pay Per Use prices the citation instead of the fetch.
More than half of crawl traffic from well-behaved bots re-fetches pages that have not changed. A signal saying nothing changed here saves the crawler compute and saves the site owner requests it never needed to serve.
Publishers in the Ceramic pilot get answer-engine reporting: which queries surfaced their content, the exact page and snippet cited, and average ranking position — Search Console-style data for AI answers.
The framing: an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → searches dozens of times to answer one question, so search becomes a utility rather than a destination, and the billable unit shifts from the crawl or the click to the outcome.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
If you're building AI search, answer-engine, or content-monetization infrastructure, this signals a real shift from pay-per-crawl to pay-per-citation economics that could change how you architect licensing and attribution for AI-consumed content.
Key quotes
“To scale the future of AI search, we need a partner with massive reach and a shared commitment to transparency and fair compensation,”
Anna Patterson
“Cloudflare allows us to easily and programmatically scale our operations. By bringing our pay-per-query model to their network, we ensure millions of content owners can seamlessly opt in to be compensated every single time their content appears in our search results.”