The old tradeoff

Some major crawlers serve both search indexing and AI training. Blocking them could also make a site harder to find in search.

“Refuse one, and you refuse the other.”

A new middle path

Cloudflare is launching Disallow AI Training, a setting meant to keep search indexing on while refusing use of site content for AI training.

Big crawlers sign on

Apple, Google, and Microsoft already honor the setting or have promised to honor it on a stated timeline.

Summaries come next

Cloudflare says AI summaries need finer controls too. A full yes or no is too blunt when the real issue is how much content appears.

“A site-wide yes or no is too blunt”

Publishers need visits

Most sites want to be found, but many depend on ads, subscriptions, or direct visitor ties. Those models break when people never arrive.

Search is welcome

Fewer than 1% of Cloudflare sites block search bots. By contrast, 17% use some method to block AI training.

Robots.txt is limited

A robots.txt file can state a preference, but it cannot prove who is crawling, why they are crawling, or stop a bot that ignores it.

Cloudflare can enforce

Cloudflare says its network can publish preferences, identify crawlers, classify their purpose, block violators, and report behavior on Radar.

Accountable crawlers

Cloudflare created an Accountable label for crawler operators that give site owners control, visibility, and search protection.

Four promises required

Operators must support training opt-outs, summary opt-outs, URL-level visibility, and assurance that training opt-outs will not hurt search ranking.

Three crawler behaviors

Cloudflare now classifies crawler controls as Search, Training, and Agent. A mixed-use crawler is one crawler doing both Search and Training.

Block means block

With Disallow AI Training available, Block and Block on pages with ads now apply to mixed-use crawlers too, including Applebot, Bingbot, and Googlebot.

Four control choices

Training controls now include Allow, Disallow AI Training, Block on pages with ads, and Block. Disallow AI Training is available only for Training.

September 15 changes

Cloudflare is deprecating Block AI Bots and Managed Robots.txt. Existing customer settings will migrate into the new Search, Training, and Agent controls.

Most users do nothing

Cloudflare says current settings carry over in almost every case. Anyone who wants mixed-use crawlers gone entirely must choose Block.

“Nothing, in almost every case.”

Ads get stricter presets

New domains get preset recommendations based on ad monetization. Sites with ads are nudged toward blocking training and limiting agents on ad pages.

Crawler details differ

Applebot and Googlebot already support training opt-outs through Applebot-Extended and Google-Extended. Bingbot support through robots.txt is targeted for early 2027.

Summaries cut both ways

Cloudflare cites data that summaries can reduce visits, while AI search referrals may convert three to five times better than traditional search referrals.

Control, not one answer

A publisher may need visitor volume, while a retailer may prefer fewer visitors with higher purchase intent. Cloudflare wants to give each site the controls to decide.