RESEARCHERS at Palo Alto Networks' Unit 42 have identified significant security vulnerabilities in the guardrails of generative AI (GenAI) tools, which are meant to prevent malicious actions like prompt injection attacks. These vulnerabilities allow large language models (LLMs), referred to as 'AI Judges,' to be manipulated into permitting violations of safety policies.
Unit 42 demonstrated an automated fuzzer called AdvJudge-Zero that can exploit these weaknesses by using subtle input sequences to influence the model's decision-making logic. This attack technique yielded a 99% success rate against various LLM architectures. Recommendations include implementing adversarial training to improve the resilience of AI models against such logic-based attacks.