Microsoft raises the bar: A smarter way to measure AI for cybersecurity

Microsoft Security Blog announces ExCyTIn-Bench, Microsoft’s newest open-source benchmarking tool designed to evaluate how well AI systems perform real-world cybersecurity investigations. It helps business leaders assess language models by simulating realistic cyberthreat scenarios and providing clear, actionable insights into how those tools reason through complex problems.

ExCyTIn-Bench evaluates AI agents in multistep, data-rich, multistage cyberattack scenarios within a simulated security operations centre in Microsoft Azure, incorporating 57 log tables from Microsoft Sentinel and related services to reflect the scale, noise, and complexity of real incidents and SOC operations.

The benchmark uses threat investigation graphs, specifically bipartite alert-entity graphs, as ground truth to support the creation of explainable question-answer pairs grounded in authentic security data, enabling analysis of strategy quality, not just final answers. Recent results show that GPT-5 leads with a 56.2% average reward, with smaller models like GPT-5-mini rivaling larger models and explicit reasoning proving crucial for complex investigations, according to Microsoft Security Blog.