CLOUDFLARE and OpenAI describe a research pilot launched in July to improve AI-driven discovery of web content. The aim is to help ChatGPT find and surface more useful pages by using signals from Cloudflare’s network to augment OpenAI’s crawlers. Early results indicate the pilot has enabled OpenAI to discover and index hundreds of millions of previously unknown pages, giving ChatGPT more sources to draw on for timely, relevant answers. The work also sheds light on what makes a page worth discovering and how AI systems process web content.
The article explains how discovery works in practice: signals help identify which pages are accessible, distinct, and helpful, while “change” and “filtering” signals manage crawl load and avoid wasting effort on non-valuable or non-existent pages. A key finding is that identifying potentially harmful pages at the page level can protect users without penalising site owners. Domain owners retain control, as a page-level signal does not grant access permissions.
The piece emphasises that this is metadata-based discovery rather than model training. Cloudflare invites site owners to use AI Crawl Control to manage how AI systems access their sites. In short, the collaboration aims to increase the breadth and relevance of AI-provided answers while reducing unnecessary crawling and maintaining safety and performance for publishers.