Cloudflare and OpenAI pilot: network signals improve AI search discovery

A search engine can only cite pages its crawler has found, and many useful pages, such as a manual for one washing-machine model or a restaurant's current menu, have few links pointing to them. Cloudflare, whose network the company says more than 20% of the web sits behind, sees which pages actually exist, which change and what kind of traffic reaches them. In July it and OpenAI started a research pilot to test whether those network signals could help OpenAI's search crawler find content for ChatGPT more efficiently. Cloudflare supplies the signals; OpenAI, the companies say, still evaluates and ranks the pages on its own.

The first findings, written jointly by Cloudflare and OpenAI staff, give one hard number. Using a cohort of about 1% of Cloudflare's network, the pilot let OpenAI discover and index hundreds of millions of web pages it had not known about. The other results are described in words rather than figures. The signals help screen out addresses that are not worth fetching: files that bots keep requesting but that do not exist, sites that redirect almost any address to their homepage, and duplicate copies of the same content. Change signals are meant to tell a crawler when a page is worth revisiting, so it does not keep downloading unchanged pages and adding load to the site. Cloudflare is also testing signals that flag a single harmful page, such as a fake login form planted on an otherwise legitimate site, without penalizing the whole domain; it says early results suggest these can help keep such pages out of an index.

Cloudflare's announcement on X promised AI search that is smarter, fresher and safer, but much of that is still a plan. The post itself notes that finding new pages can at first increase crawling while they are evaluated, and it lists faster and more reliable change signals, better page selection and measuring what happens after a page is discovered as the next phase, including whether better discovery brings site owners real business results. Cloudflare also says a signal about a page does not grant permission to access it, that the research uses metadata about the web rather than sharing page content for model training, and that site owners can manage AI crawler access with its AI Crawl Control tool.