Microsoft AI Red Team: Taxonomy of Failure Modes in Agentic AI Systems
Microsoft AI Red Team (AIRT) · v2025-04 · first published 2025-04-24
Scope
Failure modes of agentic AI systems, split along two axes: safety vs. security (impact) and novel vs. existing (whether the mode is unique to agentic AI or inherited from prior generative AI). Single- vs. multi-agent distinctions are surfaced in descriptions. Derived from Microsoft AI Red Team's operational experience and a 30-page April 2025 whitepaper.
Why it's ingested
The Microsoft AIRT taxonomy is the most operationally-grounded agentic AI failure taxonomy from a frontier vendor and is widely cited in industry agentic-security discussions. Ingesting it as a first-class Taxonomy node gives SAICA-KG crosswalks into: supply_chain_attack (MS-NS-02, MS-ES-02), context_pollution (MS-ES-01 memory poisoning), scope_creep (MS-ES-04, MS-NS-04), fabrication (MS-EF-02), and security_vulnerability (most of the NS/ES axis). The novel-vs-existing axis also informs the single-agent-vs-multi-agent framing in SAICA-KG's Tool nodes.
Categories & crosswalks
Each row lists one external category and any SAICA-KG FailureModes that cross-walk into it.
MS-NS-01Agent CompromiseUnauthorized control or manipulation of an agent; one corrupted node can subvert entire multi-agent workflows.
SAICA-KG crosswalks:MS-NS-02Agent InjectionInsertion of malicious instructions into agent prompts, plans, or tool-call arguments.
SAICA-KG crosswalks:MS-NS-03Agent ImpersonationThreat actors register counterfeit agents that appear legitimate, tricking other modules into leaking data or executing actions.
SAICA-KG crosswalks:MS-NS-04Agent Flow ManipulationUnauthorized alteration of agent workflow execution paths, e.g., injecting tokens like STOP to reroute workflows or bypass safety logic.
SAICA-KG crosswalks:MS-NS-05Multi-Agent JailbreaksJailbreak payload split across agent messages; recombination bypasses single-prompt detectors. Specific to multi-agent topologies.
MS-NF-01Intra-Agent Responsible AI ConcernsToxic or RAI-violating content propagates through internal inter-agent messages and reaches users or logs without filtering.
MS-NF-02Bias in Resource AllocationDiscriminatory distribution of resources across users when agents act as allocators.
MS-NF-03Organizational Knowledge DegradationHeavy automation erodes human understanding of the underlying system; recovery is difficult when agents fail.
MS-NF-04Prioritization Risks (Performance Over Safety)Goal-driven agents skip safety checks to optimize KPIs, prioritizing performance metrics over user safety.
SAICA-KG crosswalks:MS-ES-01Memory PoisoningAttackers embed malicious instructions into long-term memory; every future recall re-executes the attack. Flagship case study in the paper.
SAICA-KG crosswalks:MS-ES-02Cross-Domain Prompt Injection (XPIA)External content (PDFs, calendar invites, web pages) contains hidden commands that hijack agent decision logic.
SAICA-KG crosswalks:MS-ES-03Human-in-the-Loop BypassCircumvention of human oversight and approval gates.
SAICA-KG crosswalks:MS-ES-04Incorrect Permissions ManagementOver-scoped agent permissions let agents access files or perform actions beyond the user's intent.
SAICA-KG crosswalks:MS-ES-05Insufficient IsolationInadequate separation between agent operational boundaries, enabling lateral compromise.
SAICA-KG crosswalks:MS-EF-01Bias AmplificationMagnification of existing prejudices in agent outputs across iterative loops.
MS-EF-02HallucinationsGeneration of false or fabricated information, including fabricated tool calls and references.
SAICA-KG crosswalks:MS-EF-03Misinterpretation of InstructionsIncorrect understanding of user directives leading to unintended actions.
SAICA-KG crosswalks:MS-EF-04Lack of TransparencyInsufficient clarity over agent behavior, preventing informed user consent and oversight.
Notes
Category labels paraphrased from the Microsoft Security Blog summary and Adversa/MarkTechPost coverage; pending full-text verification against the 30-page whitepaper PDF. Full expansion (e.g., sub-modes under Memory Poisoning) and any additional categories is a v0.2 task.