SAICA Supervision Index
How well do popular open-source projects guard against the ways AI coding agents break things — fabricated APIs, vulnerable code, runaway loops, broken tests, and seven other common failure modes? We audit each repo weekly and grade A–F.
This page lists repos we audit. For the catalog of supervision tools you can install in your own stack, seeTool Coverage.
Last updated May 2, 2026 · 41 repos audited
What this measures
For each repo, we look at what supervision tools are already installed — Dependabot, Semgrep, pre-commit hooks, evaluation suites, observability stacks, sandboxes, and so on — and check which of the 11 known AI-coding failure modesthey actually defend against.
A high grade means the repo has layered, evidence-backeddefenses across the failure modes that matter most. A low grade doesn't mean the project is bad — it means an AI agent let loose on its codebase has fewer guardrails than it should.
How rows are ranked
Repos are grouped by category — agent frameworks, web frameworks, data libraries, AI/ML tooling, and so on — because the categories aren't comparable head-to-head (a RAG framework and a web framework face different failure modes). Within each category, repos are sorted by their supervision-coverage score (higher = more failure modes guarded, weighted by severity). The top three in each category earn a 🥇 / 🥈 / 🥉.
These are audit targets, not recommended supervision tools. A repo's position reflects how well its own development workflow defends against AI-coding failure modes — not whether the repo itself supervises other agents. For supervision tools, seeTool Coverage.
The full methodology — exactly how we map evidence to scores — is at the bottom of the page.
Agent frameworks5 repos
Score- 63.9%9/11 failure modes guarded
- 63.9%9/11 failure modes guarded
- 51.1%9/11 failure modes guarded
- 42.0%7/11 failure modes guarded
- 0.0%0/11 failure modes guarded
Data & ORM libraries6 repos
Score- 58.5%9/11 failure modes guarded
- 58.5%9/11 failure modes guarded
- 56.3%9/11 failure modes guarded
- 51.6%8/11 failure modes guarded
- 13.6%3/11 failure modes guarded
- 13.6%3/11 failure modes guarded
Web frameworks9 repos
Score- 58.5%9/11 failure modes guarded
- 50.3%7/11 failure modes guarded
- 35.6%3/11 failure modes guarded
- 35.6%3/11 failure modes guarded
- 35.6%3/11 failure modes guarded
- 35.6%3/11 failure modes guarded
- 33.4%3/11 failure modes guarded
- 31.4%4/11 failure modes guarded
- 15.8%1/11 failure modes guarded
AI / ML tooling8 repos
Score- 56.2%9/11 failure modes guarded
- 54.4%7/11 failure modes guarded
- 9/11 failure modes guarded
- 5/11 failure modes guarded
- 38.0%5/11 failure modes guarded
- 34.7%4/11 failure modes guarded
- 32.9%6/11 failure modes guarded
- 3/11 failure modes guarded
Infra & DevOps8 repos
Score- 42.0%5/11 failure modes guarded
- 42.0%5/11 failure modes guarded
- 42.0%5/11 failure modes guarded
- 38.7%5/11 failure modes guarded
- 24.6%4/11 failure modes guarded
- 13.6%3/11 failure modes guarded
- 2/11 failure modes guarded
- 2/11 failure modes guarded
Language runtimes5 repos
Score- 39.8%5/11 failure modes guarded
- 34.4%5/11 failure modes guarded
- 32.9%6/11 failure modes guarded
- 15.8%1/11 failure modes guarded
- 0.0%0/11 failure modes guarded
41 repos audited across 6 categories. Top three in each category get a 🥇 / 🥈 / 🥉. Scale: 0% (no supervision detected) → 100% (every priority failure mode covered with strong evidence).
How we score a repo
Three steps. None involve us looking at code quality — only whatdefenses you've installed against AI-coding-agent failures.
- Detect what's already there. We scan the repo for roughly two dozen common supervisors — Dependabot, Semgrep, pre-commit hooks, GitHub Actions security workflows, evaluation configs (promptfoo, deepeval, ragas), observability libraries (langfuse, helicone, opik), sandboxes (e2b, daytona), and others. Each detection comes with the file we matched on, so it's auditable.
- Map detections to failure modes. Each tool in our knowledge graph declares which of the 11 failure modes it defends against (e.g. Dependabot → dependency_blindness; Semgrep → security_vulnerability). We give partial credit for "declared coverage," more for "rationale-backed," most for "citation-backed in a paper or independent eval."
- Weight, sum, grade. Failure modes aren't equal.scope_creep is the most-reported issue (high weight);obsolescence is real but lower-impact. Each repo's per-failure-mode score is multiplied by that weight, summed, and normalized to a percentage. We add a small bonus whentwo or more control paradigms(prevention + detection, e.g.) cover the same failure mode — layered defense beats single-layer.
Grades thresholds: A ≥ 80% · B ≥ 60% ·C ≥ 40% · D ≥ 20% · F < 20%. Calibrated to be slightly punishing — most production repos today land in C/D, an A should feel earned.
Technical formula (for the curious)
Per failure mode f, the coverage score isc(f) ∈ [0, 1]: tier 1 → 0.40, tier 2 → 0.70, tier 3 → 1.00, plus+0.10 when ≥2 control paradigms cover the same FM (capped at 1.0).
Aggregate, weighted by per-FM priority (likelihood × impact, see data/failure_mode_priorities.yml):
score = Σ priority(f) × c(f) / Σ priority(f)
Source of truth:pipeline/saica_index/score.py. Per-FM priorities:data/failure_mode_priorities.yml.