Infosecurity
Echo Finds AI Models Agree on Just One of 121 Code Flaws
Three frontier AI models were tested on the same open-source code to audit potential software flaws, and their findings showed minimal agreement. Of 121 distinct issues identified, only one was flagged by all three: Anthropic's Claude Mythos 5.1, OpenAI's GPT-5.6 Sol, and Z[.]ai's GLM-5.3 each logged just a single overlapping finding.
Echo, the security firm conducting the test, disclosed that the models evaluated two Maven libraries and one npm package using its own hunting framework and the same validation process, with GPT-5.6 Sol accessed via Daybreak, OpenAI’s vetting interface for security teams. The majority of issues came from a single model, with 83% (100) of the 121 arising from one model and 20 from the other two.
In total, nine runs produced 446 initial findings, which were cleaned up to 152 after deduplication. Of these 152, 74 were rated High or Critical, and 55 advanced to automated proof‑of‑concept testing after duplicates were removed. Forty‑two findings passed validation, though Echo cautioned that these may not represent distinct bugs, since an issue flagged by more than one model could be credited to each.
The exercise was described as a controlled comparison rather than a comprehensive benchmark, and validation did not amount to confirmation from the projects’ maintainers. The projects remained unnamed, with technical details withheld until their maintainers were notified.