#robots.txt

1 story

#ai crawl control #ai summaries #ai training #applebot #bingbot #bot management #cloudflare #googlebot #microsoft #mixed-use crawlers #search crawlers

Cloudflare splits search from AI training
Cloudflare’s new setting lets sites stay visible in search while refusing AI training by mixed-use crawlers.
Cloudflare Blog · 3m