From Pymnts · PYMNTS · · 1m 20s
Cloudflare lets sites block AI training, not search
A new Cloudflare setting gives publishers a cleaner way to say yes to search crawlers and no to AI training.
In brief
A new Cloudflare setting gives publishers a cleaner way to say yes to search crawlers and no to AI training. Cloudflare is pushing crawler operators to separate search access from AI training so site owners can keep visibility without giving up control. Originally reported by Pymnts.
A cleaner crawler choice
Cloudflare launched Disallow AI Training, a setting that lets website owners stay visible in search while blocking crawlers from using their content to train AI systems.
The old tradeoff
Mixed-use crawlers created a bind. Blocking them from AI training also meant blocking them from search indexes, cutting off discovery along with unwanted reuse.
Access with conditions
With the new setting, mixed-use crawlers can reach a site only if they respect the owner's refusal to allow AI training on that content.
Big firms line up
Cloudflare says Apple, Google and Microsoft have either met the setting's criteria or committed to doing so. Other companies already split search crawlers from training crawlers.
Control becomes the pitch
CEO Matthew Prince framed the move as a way to keep search open while giving publishers and businesses more say over how their work is used.
“preserving the openness that makes search valuable”
Bots raise the stakes
PYMNTS Intelligence found nearly 90% of organizations struggle with bot activity, as AI agents and malicious bots make old defenses harder to trust.
What matters most
Cloudflare is pushing crawler operators to separate search access from AI training so site owners can keep visibility without giving up control.
