Threat modelling AI apps evolve as prompts create risk surfaces

Threat modeling AI applications is about proactively identifying, assessing, and addressing risk in AI systems, recognising that we cannot predict every misuse or emergent behaviour but can design to reduce the likelihood and impact of failures or adversarial exploits.

According to Microsoft Security Blog, AI changes threat modelling by introducing nondeterminism, instruction‑following bias, and system expansion through tools and memory, which together create new attack surfaces that don’t map neatly onto traditional models. The article emphasises starting with assets, not attacks, listing user safety, trust, privacy, data integrity, and the integrity of instructions and actions as central concerns.

It also highlights the need to understand the actual system, including how prompts, memory, data sources, tools, and user interactions interact, since the prompt assembly pipeline becomes a first‑class security boundary. Finally, it argues for architectural mitigations, strong observability, and threat modelling as an ongoing discipline shared across engineering, product, and design to manage risk at scale and maintain trust over time.