# Cloudflare splits search from AI training > Cloudflare’s new setting lets sites stay visible in search while refusing AI training by mixed-use crawlers. - Title: Cloudflare splits search from AI training · Cloudflare Blog - Summary: Cloudflare’s new setting lets sites stay visible in search while refusing AI training by mixed-use crawlers. Site owners no longer have to choose between… - Keywords: cloudflare, ai training, mixed-use crawlers, search crawlers, robots.txt, bot management, ai crawl control, applebot, googlebot, bingbot, microsoft, ai summaries - Source: Cloudflare Blog — https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers - Published: 2026-09-15T13:00:00+00:00 - Read time: 3 min - Topics: cloudflare, ai training, mixed-use crawlers, search crawlers, robots.txt, bot management, ai crawl control, applebot, googlebot, bingbot, microsoft, ai summaries ## The old tradeoff Some major crawlers serve both search indexing and AI training. Blocking them could also make a site harder to find in search. > Refuse one, and you refuse the other. ## A new middle path Cloudflare is launching Disallow AI Training, a setting meant to keep search indexing on while refusing use of site content for AI training. ## Big crawlers sign on Apple, Google, and Microsoft already honor the setting or have promised to honor it on a stated timeline. ## Summaries come next Cloudflare says AI summaries need finer controls too. A full yes or no is too blunt when the real issue is how much content appears. > A site-wide yes or no is too blunt ## Publishers need visits Most sites want to be found, but many depend on ads, subscriptions, or direct visitor ties. Those models break when people never arrive. ## Search is welcome Fewer than 1% of Cloudflare sites block search bots. By contrast, 17% use some method to block AI training. ## Robots.txt is limited A robots.txt file can state a preference, but it cannot prove who is crawling, why they are crawling, or stop a bot that ignores it. ## Cloudflare can enforce Cloudflare says its network can publish preferences, identify crawlers, classify their purpose, block violators, and report behavior on Radar. ## Accountable crawlers Cloudflare created an Accountable label for crawler operators that give site owners control, visibility, and search protection. ## Four promises required Operators must support training opt-outs, summary opt-outs, URL-level visibility, and assurance that training opt-outs will not hurt search ranking. ## Three crawler behaviors Cloudflare now classifies crawler controls as Search, Training, and Agent. A mixed-use crawler is one crawler doing both Search and Training. ## Block means block With Disallow AI Training available, Block and Block on pages with ads now apply to mixed-use crawlers too, including Applebot, Bingbot, and Googlebot. ## Four control choices Training controls now include Allow, Disallow AI Training, Block on pages with ads, and Block. Disallow AI Training is available only for Training. ## September 15 changes Cloudflare is deprecating Block AI Bots and Managed Robots.txt. Existing customer settings will migrate into the new Search, Training, and Agent controls. ## Most users do nothing Cloudflare says current settings carry over in almost every case. Anyone who wants mixed-use crawlers gone entirely must choose Block. > Nothing, in almost every case. ## Ads get stricter presets New domains get preset recommendations based on ad monetization. Sites with ads are nudged toward blocking training and limiting agents on ad pages. ## Crawler details differ Applebot and Googlebot already support training opt-outs through Applebot-Extended and Google-Extended. Bingbot support through robots.txt is targeted for early 2027. ## Summaries cut both ways Cloudflare cites data that summaries can reduce visits, while AI search referrals may convert three to five times better than traditional search referrals. ## Control, not one answer A publisher may need visitor volume, while a retailer may prefer fewer visitors with higher purchase intent. Cloudflare wants to give each site the controls to decide. ## Key takeaway Site owners no longer have to choose between search traffic and blocking AI training, but AI summaries are the next fight.