Red-team-first hardening
confident-ai-deepteam (adversarial scenarios) → guardrails-ai (runtime validators) → human review gate. Red-team scenarios double as runtime rules.
Description
Composition for security- and safety-sensitive coding-agent products. The motivating observation: prompt-injection incidents (copilot-prompt-injection-github-comments-2025, msft-airt-prompt-injection-summary-email-2025) show that failures reach production because teams treat red-teaming as a pre-launch event rather than a continuous discipline. Stage 1: confident-ai-deepteam generates adversarial scenarios (injection, jailbreak, exfiltration) against the agent. Failing scenarios are both fixed and encoded as structured rules. Stage 2: guardrails-ai enforces those rules at runtime as input/output validators on every agent invocation. Stage 3: a human review gate covers the residual — novel attack classes that neither the scenario suite nor the guardrails predict.
Stack
Ordered composition. Each stage consumes the output of the one before it.
- Stage 1DeepTeam
Open-source LLM red-teaming framework from Confident AI with 40+ attack vulnerabilities.
- Stage 2Guardrails AI
Output validation framework with a catalog of validators and reask-on-failure loops.
Targets failure modes
- Security vulnerability — Insecure patterns, injection flaws, weak cryptography, secret leakage.
- Context pollution — Trajectory drift, hallucination spirals, overlong or off-task context.
Notes
Human review is not a KG-node; stack is length 2. `case_studied` because confident-ai and guardrails-ai each publish case studies of this pattern in isolation; no joint benchmark yet.