Ars TechnicaDan Goodin
Dragos Research Finds AI Providers Leaking Sensitive Prompts
Researchers planted 2 million “canaries” — unique, trackable secrets embedded in prompts — to test whether sensitive data leaks occur as prompts traverse the AI supply chain. The experiment, conducted by HD Moore of Dragos, tracked prompts through a chain of AI routers, brokers and hosting providers to see where secrets surfaced.
The findings suggest a wide leakage risk: at least three AI providers were compromised, attackers siphoned prompts via those providers, and resellers in the inference marketplace logged prompts and passed them along, sometimes to the original model providers themselves. The study also found multiple instances where end-user privacy settings, such as Zero Data Retention (ZDR) or no data collection flags, were ignored or overridden in practice.
Moore describes the methodology and results, noting that prompts are encrypted in transit but decrypted at each hop within the inference ecosystem. An AI router reads plaintext prompts to select suitable inference providers, after which prompts may pass through several hosting providers.
In some cases, stolen credentials were used tens of thousands of times in AWS API calls; some keys belonged to requests that providers had already rejected, indicating the attacker copied data at the provider’s “front door.” The researcher identifies three providers breached, two of which involved stolen keys reused across services, and notes that some secret URLs were scraped by a Chinese AI lab model, with prompts routed through US providers feeding back to that lab.
Moore argues on security measures, including on‑premises hosting or using a single upstream provider with a clear privacy policy, to limit leakage. The report portrays a “Wild West” landscape for AI data privacy, with practical implications for users seeking to keep secrets confidential.