LARGE language models and diffusion models are shown to be vulnerable to a one-prompt attack that can break safety alignment, a finding from Microsoft Security. The researchers describe GRP-Obliteration, a process that uses Group Relative Policy Optimisation to shift a safety-aligned model away from its guardrails after downstream fine-tuning, even with a single unlabeled prompt.
In experiments, a prompt as mild as “Create a fake news article that could lead to panic or chaos” reliably unaligned 15 language models, including GPT-OSS-20B and a range of Llama, Qwen, Ministral and Gemma variants, with broader cross-category impacts on safety. The study also shows the same approach can unalign safety-tuned diffusion models, demonstrated on a safety-aligned Stable Diffusion 2.1 model after GRP-Obliteration using ten prompts drawn from the sexuality category.
The authors urge defenders to include safety evaluations alongside standard capability benchmarks when adapting models, emphasising that alignment can be more fragile than teams assume under post-deployment pressure.