Coverage Matrix
Tool × Failure-Mode evidence
Which of the supervision tools in the KG address which failure modes, and how strong is the evidence for each mapping? Cell color encodes anevidence tier for that specific (tool, failure-mode) pair— not tool popularity. GitHub stars, a crude popularity proxy, are relegated to the row labels; letting them drive color would paint a popular tool's whole row the same shade regardless of fit, which is the opposite of what this figure should surface.
This page lists supervision tools you can install.For weekly audits of popular repos and how supervised they are, see the SAICA Index.
Tiers
- 0 — tool does not list this failure mode in
addresses_failure_modes. - 1 — declared mapping: the tool's YAML asserts it addresses this FM, but nothing more.
- 2 — rationale mentions it: the tool's
inclusion_rationaleactually discusses this FM (id, synonym, or phrase match). A per-mapping signal, not a per-tool one. - 3 — external citation: tier 2 plus the tool has a paper / dataset / documentation reference in
cited_in,documented_in, orevaluated_on.
Generates instructions for your coding agent (Claude Code, Cursor, Replit Agent, …) to integrate exactly the tools in the selected view, then wire them into SAICA's commit checks: a git hook and a CI job run them on every commit, and a local dashboard shows what they catch over time. The agent inspects your repo and proposes a plan before changing anything. Coverage shown here isdeclared, not verified protection.
| Tool | Stars | Breadth |
|---|---|---|
| Loading… | ||
Four views, four questions
- "Optimal — 3 tools" (default) — greedy weighted set cover capped at 3, weighted by failure-mode likelihood × impact times tool reliability. Best "responsible kit"; covers most of the high-priority FMs with the smallest practical stack. Answers: "if I install three things, what should they be?"
- "Minimum — 1 tool" — the single highest-priority × reliability tool. Best starter pick. Answers: "if I install only one thing, what?"
- "Full / MECE" — minimum number of tools needed to cover all 11 failure modes (no pad). Typically 4–5 tools. Answers: "what's the smallest set that covers everything?"
- "All tools with ≥1 mapping" — every tool that declares any FM. Useful when you're hunting for a specific tool (Context Hub, Semgrep, Renovate, etc.) that doesn't make the top views.
Almost every filled cell is tier 1 right now. Only a handful of tier-2 cells (rationales that explicitly name the FM), and very few tier-3 (citation-backed) cells. The KG honestly admits that today it holds declarations more than evidence — closing that gap is exactly what the citation / crosswalk pipeline is built to do.
Methodology notes
Selection is now likelihood × impact × reliability. Per-FM likelihood and impact are stored in data/failure_mode_priorities.yml (hybrid: KG tool-coverage prior + editorial calibration against Shah 2026 / DAPLab evidence). Reliability is a bounded combiner of log-stars, github-trending bump, citation count, and maturity status. The greedy set-cover algorithm at every step picks the tool maximisingΣ priority(fm) over newly-covered FMs × tool reliability.
Coding-agent peers are filtered out per asker (the user has a coding agent already; we recommend supervisors, not peers). The comfyui blocklist excludes role-ambiguous tools.
Reproduce with .venv/bin/python research/plot_tool_fm_heatmap.py. Outputs live under research/figures/ (static PNG/PDF for the paper) andsite/public/tool_fm_heatmap.json (data for this page).