Cloudflare Lets Sites Block AI Training Without Losing Search Traffic

Cloudflare has announced a new “Disallow AI Training” setting for its Bot Management and AI Crawl Control services, allowing website owners to refuse AI training while keeping their content available to search crawlers. The setting publishes the relevant preference in `robots.txt` and is designed for mixed-use crawlers that support both search indexing and AI training.

Cloudflare says Applebot, Googlebot and Bingbot are treated as “Accountable” because their operators provide, or have committed to provide, controls and transparency around content use. Cloudflare reports that fewer than 1% of its sites block search bots, while 17% use some mechanism to block training.

The changes take effect on 15 September 2026. Existing settings will generally migrate automatically: previous training selections of “Block” or “Block on pages with ads” become “Disallow AI Training”. Selecting “Block” will now block mixed-use crawlers entirely, including search access. The new control blocks separate training-only crawlers operated by Amazon, Anthropic, Meta and OpenAI without affecting search. Cloudflare is also deprecating “Block AI Bots” and Managed Robots.txt in favour of separate Search, Training and Agent controls.

There are important operator-specific limitations. Google supports `Google-Extended`, reporting tools and a control for generative search, while Apple supports `Applebot-Extended` and related page directives. Microsoft currently supports `NOARCHIVE` and removal tools; its robots.txt no-training mechanism is targeted for early 2027, so Cloudflare’s setting will not automatically convey that preference to Bing until then. Cloudflare says it plans to offer more granular control over AI summaries by early next year.