Skip to main content
SAICA-KG

Recipes

Red-team-first hardening

confident-ai-deepteam (adversarial scenarios) → guardrails-ai (runtime validators) → human review gate. Red-team scenarios double as runtime rules.

case studiedpolyglot

Description

Composition for security- and safety-sensitive coding-agent products. The motivating observation: prompt-injection incidents (copilot-prompt-injection-github-comments-2025, msft-airt-prompt-injection-summary-email-2025) show that failures reach production because teams treat red-teaming as a pre-launch event rather than a continuous discipline. Stage 1: confident-ai-deepteam generates adversarial scenarios (injection, jailbreak, exfiltration) against the agent. Failing scenarios are both fixed and encoded as structured rules. Stage 2: guardrails-ai enforces those rules at runtime as input/output validators on every agent invocation. Stage 3: a human review gate covers the residual — novel attack classes that neither the scenario suite nor the guardrails predict.

Stack

Ordered composition. Each stage consumes the output of the one before it.

  1. Stage 1
    DeepTeam

    Open-source LLM red-teaming framework from Confident AI with 40+ attack vulnerabilities.

  2. Stage 2
    Guardrails AI

    Output validation framework with a catalog of validators and reask-on-failure loops.

Targets failure modes

Notes

Human review is not a KG-node; stack is length 2. `case_studied` because confident-ai and guardrails-ai each publish case studies of this pattern in isolation; no joint benchmark yet.

View raw YAML on GitHub →