AI Browsers That Beat Paywalls by Imitating Humans

AI-powered browsers such as OpenAI’s Atlas, Perplexity’s Comet and Microsoft’s Copilot mode in Edge are described as capable of bypassing paywalls by mimicking legitimate human users, a development that poses new challenges for publishers seeking to protect paywalled content.

The piece notes that Atlas and Comet managed to retrieve the full text of a nine-thousand-word subscriber-exclusive article from the MIT Technology Review, whereas ChatGPT and Perplexity’s standard interfaces reportedly could not access the same content due to blocking by the publisher’s crawlers.

It explains that to a website, Atlas’s AI agent can appear indistinguishable from a person using a normal Chrome session, complicating crawler blocking strategies which rely on digital IDs and the Robots Exclusion Protocol. The article distinguishes paywall approaches, with some outlets using client-side overlays and others using server-side paywalls that still permit access once credentials are verified, albeit potentially through AI-assisted workflows.

It notes that OpenAI states that by default it does not train its models on content encountered in Atlas unless users opt into browser memories, according to OpenAI.