Researchers break OpenAI guardrails

According to Malwarebytes, researchers broke OpenAI’s guardrails after the firm released a toolkit at its DevDay on 6 October for building AI agents using its ChatGPT technology, including a tool called AgentKit. The Guardrails aim to stop agents from harmful activities, such as giving instructions to produce anthrax spores, but researchers found ways to bypass them.

HiddenLayer, an AI security company, successfully cracked the jailbreak protections by using a prompt injection attack that persuaded the guardrails’ underlying large language model to lower its confidence score and accept normally unacceptable prompts. They also defeated the prompt-injection detector built into OpenAI’s Guardrails offering.

The piece notes this is part of an ongoing arms race between attackers and defenders in the AI security space, with previous techniques such as Policy Puppetry and comparisons of guardrail effectiveness across platforms highlighted. Danny Bradbury’s article, posted on 13 October 2025, emphasises the need to remain cautious about what users tell an AI assistant, since even seemingly private conversations can be at risk.