Cloudflare and OpenAI are collaborating on a pilot project to enhance the freshness of AI search results.

      Cloudflare has dedicated the past year to fortifying websites against AI crawlers, and now it aims to assist one in crawling more effectively. On Wednesday, the company announced it is collaborating with OpenAI on a research pilot project to determine if its network data can enhance the accuracy of AI search results.

      Cloudflare monitors over 20% of the internet, which enables it to observe real-time changes in web pages and user traffic patterns. This pilot will provide OpenAI's search and answer system with key signals, including content freshness and traffic quality.

      The importance of freshness

      The reliability of AI responses depends heavily on the pages they are drawn from. If an index is outdated, chatbots may return outdated or incorrect information. OpenAI seeks to leverage Cloudflare’s live web insights to ensure that ChatGPT identifies new content more quickly.

      In addition, OpenAI has established content agreements to enrich ChatGPT. However, this pilot aims to address a distinct challenge: content discovery rather than licensing.

      “Having current information is crucial for providing accurate responses to ChatGPT users,” stated Nick Ryder, OpenAI’s VP of research. He described the test as a method for more efficient content discovery.

      A significant shift

      This initiative appears to contrast with Cloudflare's recent actions. Last year, the company allowed websites to prevent AI crawlers from accessing their content unless AI companies compensated publishers. It also developed a browser protocol to distinguish between humans and bots, promoting the principle: your content, your rules.

      This pilot, however, takes a different approach. Instead of restricting access to content, Cloudflare is now facilitating AI engines’ access to it. CEO Matthew Prince framed this as a matter of efficiency, aiming to “enhance AI search efficiency and help users obtain quality answers more rapidly.”

      Why it matters

      The search landscape is transitioning from link-based to answer-oriented results. Publishers are already concerned about losing traffic to chatbots that summarize their content—a concern that has led regulators to enforce opt-out measures. Cloudflare now occupies a unique position, operating on both sides of the issue. It provides tools to block crawlers while also exploring the mechanisms to support them. The outcome of this pilot could influence who has access to the open web and the conditions under which that access is granted.