Dark ReadingNate Nelson
Researchers hide sleeper payloads in open weight AI models
Researchers have described a technique to hide small malicious payloads inside open-weight AI models, raising concerns about the safety of downloading neural networks from the open internet. The approach, dubbed by some as a “sleeper agent,” can place a payload in a tiny fraction of a model’s neurons, with the payload only activating under a specific trigger.
Microsoft researchers have described a variant that targets “dormant” neurons late in a model’s processing cycle, arranging them to fire sequentially and remain largely inactive until fed a trigger. Briland Hitaj, a senior AI security researcher at Confidencial, notes that the fundamental vulnerability lies in how neural networks are stored and used rather than in a traditional software bug, since weights are rarely inspected in the same way as code.
The piece explains two historical strands of this research. In 2022, Hitaj and colleagues introduced MaleficNet, which encoded malware across many network weights using a spread-spectrum technique. The article explains that the more recent Microsoft approach aims to compress a payload into under 10 neurons, to be decoded when a trigger is provided by user input, tool output, or even a simple date.
Mancini, a professor at Sapienza University of Rome and co-author of the MaleficNet work, advocates a defence-in-depth stance for open-weight AI security, including validating models against trusted baselines, diffing deployed weights from originals, and restricting model access, credentials, and network connections. The piece emphasises monitoring for anomalous model behaviour at the system boundary as part of practical detection.