blog.cloudflare.com 7/1/2026, 3:22:14 PM · external

New AI bot defaults block Agent, Training crawlers on ad pages

New AI bot defaults block Agent, Training crawlers on ad pages
CyberSIXT Evidence Panel Source marked as original reporting

THE article discusses new AI traffic management options for website owners, reflecting on changes that have occurred since the initial declaration of 'Content Independence Day'. It introduces a more nuanced taxonomy of bots, classifying them into three main use cases: Search, Agent, and Training. These classifications allow website owners to manage AI traffic more effectively. Upcoming default settings for new domains will block Agent and Training crawlers on ad-displaying pages while allowing Search by default.

Additionally, BotBase, a new tool for Enterprise customers, will provide enhanced visibility of bot behaviors. The article emphasizes the need for transparency in bot behavior and empowers site owners to set terms for their content's usage.

View full article

Article by CyberSIXT