Cloudflare sets a September deadline for AI crawlers to make payments.

      Cloudflare has given the AI industry a deadline. Starting in September, it will block crawlers that collect content for AI training purposes. Any page that contains advertisements will be restricted, unless the website owner opts otherwise. The main message is clear: stop providing free access to web content.

      The company handles a significant portion of global web traffic and announced this change on Wednesday. From September 15, new Cloudflare sites will continue to allow search engines to index their content, but they will by default block AI training and agents from accessing any pages with ads.

      This rule also affects “mixed-use” crawlers, which are bots that perform search, training, and agent functions simultaneously. If a crawler does not permit site owners to differentiate these functions, it will be blocked from ad-supported pages.

      These default settings will apply to new customers as well as to new sites from existing customers. They will also include every free user who has not modified their settings. However, site owners can always allow bots back in via their dashboard. The default stance has shifted: content that generates revenue is now off-limits to AI unless the owner permits it.

      Why is this happening now? Cloudflare’s case is backed by a striking statistic. Automated bots now account for over half of all web traffic, a milestone the company claims was reached sooner than anticipated. CEO Matthew Prince stated that the majority of internet traffic is now generated by non-human entities. He emphasized that Cloudflare “must go further and act faster to foster a sustainable ecosystem.”

      The underlying issue is one that publishers understand well. Most websites aspire to be included in AI responses, just as they want to rank well in search results. However, the same crawling often feeds models that respond directly to users, bypassing visits and ad revenue entirely.

      Cloudflare specifically called out the “world’s largest search engine,” implicitly referring to Google. Its Googlebot combines indexing with AI training, which grants Google approximately double the data access compared to competing AI companies. Blocking this bot poses the risk of disappearing from search results, and the same challenge applies to Microsoft’s Bing and Apple’s Applebot.

      Blocking is merely one part of the strategy. Cloudflare is transforming last year’s “Pay Per Crawl” model into a “Pay Per Use” structure. It will now compensate publishers when their content contributes to an AI-generated response, rather than only paying when the content is retrieved. Early partners include AI search companies Ceramic.ai and You.com.

      Cloudflare is also introducing a dashboard for publishers to track which bots utilize their content and the minimal traffic these firms return. This concept is termed Answer Engine Optimization, a new term for the AI era akin to SEO.

      This shift is occurring in a market that is already moving in that direction. A wave of startups is offering tools to help brands remain visible within chatbots, betting that GEO will replace SEO. Cloudflare aims to be the foundational support for this transformation.

      The situation for publishers is dire. AI-generated responses are diminishing the clicks that sustain the web, keeping users on Google or within chatbots instead of leading them to the original content creators. One study indicated that Google’s AI Overviews reduced outbound clicks by approximately 40 percent. Economists have begun to model a potential collapse of the open web if this issue is not addressed.

      It remains uncertain whether a single company can resolve this issue. Google and Apple already provide opt-out crawlers that may bypass Cloudflare’s restrictions, and competitors could find workarounds as well. Regulators are also approaching the same problem from a different perspective; for instance, the UK is compelling Google to allow publishers to opt out of AI searches without jeopardizing their ranking, and news publishers are taking legal action against OpenAI over training practices.

      Cloudflare’s action marks the most aggressive effort to ensure that AI compensates for the content it consumes. The deadline looms on September 15, and the entire web will be watching to see how the AI giants respond.