Skip to main content
SAICA-KG

Coverage Matrix

Tool × Failure-Mode evidence

Which of the supervision tools in the KG address which failure modes, and how strong is the evidence for each mapping? Cell color encodes anevidence tier for that specific (tool, failure-mode) pair— not tool popularity. GitHub stars, a crude popularity proxy, are relegated to the row labels; letting them drive color would paint a popular tool's whole row the same shade regardless of fit, which is the opposite of what this figure should surface.

This page lists supervision tools you can install.For weekly audits of popular repos and how supervised they are, see the SAICA Index.

Tiers

Generates instructions for your coding agent (Claude Code, Cursor, Replit Agent, …) to integrate exactly the tools in the selected view, then wire them into SAICA's commit checks: a git hook and a CI job run them on every commit, and a local dashboard shows what they catch over time. The agent inspects your repo and proposes a plan before changing anything. Coverage shown here isdeclared, not verified protection.

ToolStarsBreadth
Loading…

Four views, four questions

Almost every filled cell is tier 1 right now. Only a handful of tier-2 cells (rationales that explicitly name the FM), and very few tier-3 (citation-backed) cells. The KG honestly admits that today it holds declarations more than evidence — closing that gap is exactly what the citation / crosswalk pipeline is built to do.

Methodology notes

Selection is now likelihood × impact × reliability. Per-FM likelihood and impact are stored in data/failure_mode_priorities.yml (hybrid: KG tool-coverage prior + editorial calibration against Shah 2026 / DAPLab evidence). Reliability is a bounded combiner of log-stars, github-trending bump, citation count, and maturity status. The greedy set-cover algorithm at every step picks the tool maximisingΣ priority(fm) over newly-covered FMs × tool reliability.

Coding-agent peers are filtered out per asker (the user has a coding agent already; we recommend supervisors, not peers). The comfyui blocklist excludes role-ambiguous tools.

Reproduce with .venv/bin/python research/plot_tool_fm_heatmap.py. Outputs live under research/figures/ (static PNG/PDF for the paper) andsite/public/tool_fm_heatmap.json (data for this page).