Skip to content
NewsProduct launch

Cloudflare splits AI crawlers into three lanes and sets a September 15 default-block deadline

· by Pondero Newsdesk

The short version

Cloudflare launched granular controls on July 1, 2026 letting any site owner separately manage three categories of AI traffic: Search, Agent, and Training. Starting September 15, Training and Agent crawlers will be blocked by default on ad-displaying pages for all new domains.

Cloudflare splits AI crawlers into three lanes and sets a September 15 default-block deadline

The long-standing binary of "allow all bots" or "block AI bots" ended on July 1, 2026. Cloudflare replaced it with a three-lane taxonomy covering every site on its network, from free-tier blogs to enterprise CDN customers, and set a hard deadline: starting September 15, new domains on Cloudflare will have two of those three lanes blocked by default on every page that displays an ad.

What Cloudflare launched

The new controls live in the Security settings of the Cloudflare dashboard and divide AI traffic into three named categories, per Cloudflare's July 1 announcement:

Search: crawlers that index content so it can be retrieved later in answer to a query. Cloudflare's framing is that Search should be expected to return referral traffic or "other equitable compensation" to the site owner.

Agent: automated traffic acting on a human's behalf in real time. This bucket covers chat-fetch bots such as ChatGPT-User and browser-use agents such as Claude or Gemini driving a Chrome session. A human is typically waiting on the other end of an Agent request.

Training: crawlers that ingest content to train or fine-tune a model. Per Cloudflare, the key characteristic is that the site's data is "permanently absorbed into the underlying architecture of the AI."

All three controls are available to customers on the free tier, not just paying plans. Site owners set their preferences in the zone's Security settings, and can mix and match: allow Search, block Training, leave Agent on a case-by-case basis.

The September 15 deadline

The deadline is the most consequential part of the announcement for publishers who have not yet touched their Cloudflare settings.

Starting September 15, 2026, every new domain that onboards to Cloudflare will inherit these defaults on pages that display ads: Training blocked, Agent blocked, Search allowed. The logic, per the Cloudflare blog, is that an ad on a page signals "a website owner meant for a person to land there and see it." Bots that prevent that human attention from reaching the page are treated as unwelcome by default.

Existing domains are not changed automatically. Cloudflare said it will notify existing customers ahead of the deadline and said they can "easily mark" in Security settings that they want no changes before September 15.

There is one wrinkle that catches multi-purpose crawlers. Googlebot, Applebot, and BingBot each combine Search indexing with Training data collection. Under the new rules, multi-purpose crawlers are evaluated against all of their behaviors. A site owner who blocks Training will block these crawlers too, because the most restrictive applicable rule wins. Cloudflare said it "strongly encourages" bot operators to separate their crawlers into distinct user agents for each purpose.

What else shipped alongside the controls

BotBase is a new database Cloudflare is opening to Enterprise Bot Management customers. It provides a searchable directory of every tracked bot and agent classified under the taxonomy above, visible directly from the dashboard. Cloudflare described it as "a comprehensive, searchable view of our entire directory of bots" with plans to expand it into a direct control center for individual bot rules later in 2026.

Content-use signals extend the existing robots.txt Content Signals standard with a fourth field, use, that expresses how a site owner permits content to be stored and reshared. The three levels are immediate (no storage or reuse), reference (the default: index, excerpt, and link back), and full (summarize and reproduce). Customers who had already enabled Cloudflare's managed robots.txt will receive use=reference prepended automatically. A bot that abuses these signals loses its Verified status in BotBase, which removes its default-allowed treatment.

Transitive trust via the RFC 7239 Forwarded header is in early testing. When a site owner allows a specific operator, that permission carries through intermediary layers. Cloudflare offered the example Forwarded: for="openai";use="reference" as how an operator would signal both its identity and its intended content use. The mechanism targets the reality that many AI agents run through platform intermediaries rather than direct connections.

Why it matters

The September 15 deadline is the first industry-wide default that treats Training and Agent crawlers differently from Search at the infrastructure layer, not just through robots.txt signals that crawlers can ignore. Cloudflare sits in front of more than 20% of web domains globally, per Cloudflare's own network data, so a default-block that applies to new domains has immediate reach.

For publishers running on Cloudflare, the decision is no longer passive. The absence of a setting is itself a choice: existing domains with no configuration will see no change on September 15, while new domains launch with Training and Agent blocked on ad pages by default. Publishers who want Training crawlers to reach their content for AI answer-engine visibility will need to set that preference explicitly.

The three-way split also changes the risk calculation for AI labs. A crawler that claims Search behavior but also absorbs training data can no longer hide behind a single user agent without triggering block rules on sites that allow Search but block Training. That creates a compliance incentive: maintain separate user agents for each purpose or risk being treated as the most restrictive category.

For teams running agentic workflows that browse the web on behalf of users, the Agent category is the one to watch. A site that blocks Agents will refuse Claude-driven Chrome sessions, ChatGPT browse mode, and any other autonomous browsing bot, regardless of what task the human asked for. As more publishers arrive on Cloudflare with Agent blocked by default, AI product teams may need to negotiate per-site allowances or route through crawl compensation agreements.

Context and reactions

Cloudflare framed the July 1 launch as "Content Independence Day 2026," one year after its first Content Independence Day announcement in 2025, which introduced a one-click Block AI Bots option and a Pay-Per-Crawl marketplace. The 2025 announcement gave site owners a single toggle; the 2026 version gives them three separate dials with richer signal about what each type of bot does.

The broader backdrop is a documented gap between how much AI crawlers consume and how much referral traffic they return. A separate Cloudflare data post from August 2025 showed crawling by search engines and AI services had surged 24% year-over-year by mid-2025 before flattening. Pew Research Center studies cited in that report found Google AI Overviews were associated with declining click-through rates for news sites.

No major CDN or hosting provider had announced a comparable three-lane taxonomy or default-block schedule as of July 4, 2026.

What to watch next

Three milestones will determine how much this changes the crawling landscape. First, whether AI labs (OpenAI, Anthropic, Google DeepMind) update their crawler user-agent strings to cleanly separate Search, Agent, and Training behaviors before September 15, given the strong incentive Cloudflare has created. Second, whether competing CDN providers or web hosts adopt a similar taxonomy or their own defaults, which would extend the effect beyond Cloudflare's network. Third, whether publishers in verticals that depend on AI answer-engine visibility (news, e-commerce, reference content) start actively configuring Search-allowed plus Training-blocked as their default stance, which would be the scenario Cloudflare's controls are designed to enable.

Publishers using Cloudflare today can configure their preference in the zone Security settings now, ahead of any deadline.

Sources