Tech firms back robots.txt standard for AI training controls

Major technology firms have united behind a proposed AI training preference standard that publishers can declare in robots.txt files. Google, Microsoft, OpenAI, and Anthropic are backing the specification, which aims to give website administrators a transparent and scalable way to express how their content may be used to train foundational AI models.

The standard introduces a universal user-agent token, __usage_ai-training, allowing site owners to block or permit AI training across their entire domain or for specific directories. The article notes a simple example:User-Agent: __usage_ai-training Disallow: / and User-Agent: __usage_ai-training Allow: /archive/. The drive behind the move is to provide publishers with governance over proprietary content as AI systems ingest training data.

Deployment timelines are outlined as platforms plan to implement the controls in stages. Microsoft says its systems will honour these declarations from 1 February 2027, while Google plans systemwide recognition in early 2027 as engineering work wraps up. The framework is described as an internet proposal—not yet finalised—with OpenAI’s announcement and Anthropic’s crawling practices cited as examples of progress.

Crucially, the article stresses that the declaration distinguishes between pre-training data ingestion and real-time generative citations: it does not block indexing by crawlers, and publishers seeking full crawler exclusion should still use traditional robots.txt directives. Google’s guidance on AI training controls is referenced, noting that opting out can affect broader system interactions, and that the standard marks a milestone in collaborative digital data stewardship.

The piece frames the development as a step toward clearer rights management for online content, with industry players signalling readiness to adopt the rules as they mature. CVEs highlighted in related sections include CVE-2026-20051, CVE-2026-20118, CVE-2026-111357, CVE-2026-118527, and CVE-2026-120281 among others.