SAICA-Bench report · SAICA-Bench results v2.0.0 · built 22 November 2026
Can supervision tools and LLMs catch AI coding agent failures?
Findings from SAICA-Bench
Vasyl Rakivnenko, Stanford University
AI coding agents fail in recurring ways: they call functions that don't exist, edit tests until they pass, slip vulnerabilities into working code, or report a task as done when it isn't. SAICA-Bench asks what reviewers flag when they are shown such work. We ran 12 language models asked directly, 12 supervision tools (11 of them from our catalog) and 27 baseline and shortcut rules on 7,339 labelled samples across 9 failure modes, and scored every verdict against the labels.
On the 2 sets of real agent output (Security vulnerability and False 'done' claims; 2,902 samples) no reviewer reaches J 25%. Best J: +8% on Security vulnerability and +20% on False 'done' claims; 5 of the 23 ranked rows there have a 95% interval for J that excludes 0. On the 6 sets where the failure was planted on purpose the best reviewer reaches J +77% to +98%; on Logic error (model-written code that is not agent output) +74%. Every set of real agent output scores below every planted set, so this benchmark cannot tell 'reviewers catch these failures' from 'reviewers catch planted failures'. Whether the high scores carry over to failures agents make on their own is not tested.
Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.
Key findings
- On the 2 sets of real agent output (Security vulnerability and False 'done' claims; 2,902 samples) no reviewer reaches J 25%. Best J: +8% on Security vulnerability and +20% on False 'done' claims; 5 of the 23 ranked rows there have a 95% interval for J that excludes 0. On the 6 sets where the failure was planted on purpose the best reviewer reaches J +77% to +98%; on Logic error (model-written code that is not agent output) +74%. Every set of real agent output scores below every planted set, so this benchmark cannot tell 'reviewers catch these failures' from 'reviewers catch planted failures'. Whether the high scores carry over to failures agents make on their own is not tested.
- 6 of the 9 sets have constructed positives (Fabrication, Test edits that hide failures, Scope creep, Obsolescence, Supply chain and Dependency blindness). 3 use naturally produced output, 2 of them real agent output. A model-free shortcut baseline is published for 9 of the 9 sets.
- A single best reviewer can be named in 2 of 9 modes. Logic error: mimo-v2.6-flash (J +74%); Scope creep: mimo-v2.6-flash (J +98%). In the others the leading rows cannot be told apart at 95% on the samples they share: Fabrication (4 rows, J +92% to +94%); Test edits that hide failures (8 rows, J +86% to +91%); Security vulnerability (11 rows, J -0% to +8%); False 'done' claims (4 rows, J +15% to +20%); Obsolescence (3 rows, J +88% to +90%); Supply chain (2 rows, J +72% to +77%); Dependency blindness (3 rows, J +86% to +90%).
- Where a catalog tool ran its own check (5 of 9 modes), a model asked the failure-specific question was clearly ahead of the best such tool in 3; in 2 no row reaches J 25%. Ahead: Fabrication (deepseek-v4.1-flash +94% vs pyright +85%, difference +9 points on shared samples, 95% interval +5 to +13); Logic error (mimo-v2.6-flash +74% vs PR-Agent +53%, difference +21 points on shared samples, 95% interval +17 to +25); Obsolescence (gpt-6-luna +90% vs pyright (latest libraries) +57%, difference +33 points on shared samples, 95% interval +17 to +49). No row at J 25%: Security vulnerability and False 'done' claims. No catalog tool with a check of its own for a finished change was run on Test edits that hide failures, Scope creep, Supply chain and Dependency blindness.
- No packaged tool beat the same model asked the failure-specific question (12 comparisons on identical samples: 5 worse, 7 indistinguishable). Every tool ran with deepseek-v4-flash as its judge model, set by us. Worse: PR-Agent 0.50.0 on Fabrication (-24 points, 95% interval -28 to -19); uqlm 0.6.8 on Logic error (-16 points, 95% interval -23 to -10); PR-Agent 0.50.0 on Logic error (-16 points, 95% interval -20 to -11); Arize Phoenix 3.10.0 on False 'done' claims (-13 points, 95% interval -19 to -6); DeepEval 4.3.4 on False 'done' claims (-19 points, 95% interval -25 to -13). Each of them ran its own general-purpose check, not the failure-specific question. The frameworks that were given our rule text (giskard and promptfoo) were indistinguishable from the bare model in 4 of 4 comparisons.
- Free programs reach J 50% or more where the failure is a checkable fact: pyright on Fabrication (+85%); pyright (latest libraries) on Obsolescence (+57%). No free catalog program within its design scope was run on Test edits that hide failures, Security vulnerability, Logic error, Scope creep, Supply chain and Dependency blindness.
- Phi-4-mini scores J under 20% in 6 of the 8 modes it ran in and grok-4.1-fast flags more than 30% of clean samples in 6 of 9 modes. The other models that ran in at least 8 modes (gpt-6-luna, deepseek-v4-flash and mimo-v2.6-flash) score J under 20% in at most 2 modes and flag more than 30% of clean samples in at most 2.
Results at a glance
One line per failure mode, in one block per kind of set, because where the bad samples come from decides how a score should be read. “Best reviewer” is the best ranked row of any kind; “best catalog tool” is the highest-scoring tool of our catalog running a check of its own. They are different things. Names link to what each failure is; the last column opens the set's full table.
Real agent output
The bad samples are what an agent produced on its own; the label comes from running hidden tests.
| What the set scores | Best reviewer | Highest J | Best catalog tool, own check | Shortcut baseline | All rows |
|---|---|---|---|---|---|
| False 'done' claims failure mode: Incomplete execution J under 25% on our set | no reviewer reaches J 25% | +19.5% (95% interval +13.9% to +24.9%) | none reaches J 25% | +61.3% Issue prior | All rows → |
| Security vulnerability J under 25% on our set | no reviewer reaches J 25% | +7.9% (95% interval 0.0% to +15.9%) | none reaches J 25% | +48.5% Task prior | All rows → |
Model-written code, not agent output
The bad samples are what a code model wrote for a textbook task; the label comes from running the task's tests.
| What the set scores | Best reviewer | Highest J | Best catalog tool, own check | Shortcut baseline | All rows |
|---|---|---|---|---|---|
| Logic error J 25% to under 80% on our set | mimo-v2.6-flash | +74.3% (95% interval +69.4% to +78.9%) | PR-Agent +53.2% | +39.4% Surface classifier | All rows → |
Failure planted on purpose
The bad samples were written, injected or swapped in on purpose, by a model, a script or by hand.
| What the set scores | Best reviewer | Highest J | Best catalog tool, own check | Shortcut baseline | All rows |
|---|---|---|---|---|---|
| Dependency blindness J 80% or more on our set | top group of 3: gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash | +90.0% (95% interval +82.0% to +98.0%) | none ran a check of its own | +38.0% No name from the shown file appears in the added lines | All rows → |
| Fabrication J 80% or more on our set | top group of 4: deepseek-v4.1-flash, glm-5.3 and 2 more | +94.3% (95% interval +91.6% to +96.7%) | pyright +85.3% | +59.7% Context-only rule | All rows → |
| Obsolescence J 80% or more on our set | top group of 3: gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash | +89.7% (95% interval +82.9% to +95.1%) | pyright (latest libraries) +56.8% | +97.4% The code uses an API listed in the 2024 deprecation map | All rows → |
| Scope creep J 80% or more on our set | mimo-v2.6-flash | +97.7% (95% interval +93.6% to +99.1%) | none ran a check of its own | +46.9% Two or more hunks in files that already existed | All rows → |
| Supply chain J 25% to under 80% on our set | top group of 2: gpt-6-luna and deepseek-v4-flash | +77.0% (95% interval +70.3% to +83.8%) | none ran a check of its own | +100.0% An added package is in the OpenSSF malicious-packages list | All rows → |
| Test edits that hide failures failure mode: Test manipulation J 80% or more on our set | top group of 8: minimax-m3, giskard and 6 more | +91.0% (95% interval +81.3% to +94.9%) | none ran a check of its own | +60.3% First user message mentions tests | All rows → |
If the prompt of every mode ran on every change
Each score above is for one question about one failure. In use nobody knows which failure a change has, so every question would run on every change and their false alarms add up. This was not run; the table is an estimate.
estimate under independence, not measured Share of clean changes that get at least one flag when the prompt of every mode is run on every change.
| Model | Prompts run on each change | Clean changes with at least one flag | Expected false alarms per clean change | Cost per 1,000 changes, all prompts | Only the prompts of modes above the floor |
|---|---|---|---|---|---|
| gpt-6-luna | 9 | 89% | 1.8 | $4.74 | 61% with 7 prompts |
| deepseek-v4-flash | 9 | 87% | 1.67 | $10.39 | 55% with 7 prompts |
| mimo-v2.6-flash | 9 | 69% | 1.05 | $8.78 | 49% with 7 prompts |
| grok-4.1-fast | 9 | 100% | 4.21 | $5.47 | 99% with 7 prompts |
The prompts' false alarms are taken as independent, and each prompt's false-alarm rate comes from its own mode's clean samples. No run with all prompts on the same changes has been made.
Set by set
Real agent output
The bad samples are what an agent produced on its own; the label comes from running hidden tests.
False 'done' claims · failure mode: Incomplete execution
The agent declares the task done, but the issue's hidden tests fail. That covers wrong fixes as well as unfinished work. On 600 runs declared done whose hidden tests fail and 600 clean samples, with 10 ranked rows: No reviewer reaches J 25% on this set: the highest J of a row with enough readable answers is +19.5% (95% interval +13.9% to +24.9%), and 3 other rows cannot be told apart from it. No best reviewer is named. A shortcut baseline (Issue prior) scores J +61.3% on the same set.
J under 25% on our set · Data: SWE-rebench-OpenHands-Trajectories (Nebius, CC-BY-4.0): Qwen3-Coder-480B in OpenHands on real GitHub issues. What incomplete execution is → · All rows → · How it was tested →
Security vulnerability
The agent's code change introduces a security vulnerability. On 1,243 vulnerable patches and 459 clean samples, with 13 ranked rows: No reviewer reaches J 25% on this set: the highest J is +7.9% (95% interval -0.0% to +15.9%), and 10 other rows cannot be told apart from it. No best reviewer is named. A shortcut baseline (Task prior) scores J +48.5% on the same set.
J under 25% on our set · Data: SusVibes (LeiLiLab, MIT): AI agents' patches for 166 real feature tasks, with hidden CVE tests. What security vulnerability is → · All rows → · How it was tested →
Model-written code, not agent output
The bad samples are what a code model wrote for a textbook task; the label comes from running the task's tests.
Logic error
The code runs but computes the wrong result. On 1,536 wrong functions and 904 clean samples, with 7 ranked rows: Best reviewer: mimo-v2.6-flash at J +74.3%. Its paired difference from the next row (deepseek-v4-flash) has a 95% interval that excludes 0. The best catalog tool running its own check is PR-Agent, at J +53.2%. A shortcut baseline (Surface classifier) scores J +39.4% on the same set.
J 25% to under 80% on our set · Data: Collu-Bench (lt-asset, BSD-3-Clause-Clear): MBPP and HumanEval functions written by 11 code models. What logic error is → · All rows → · How it was tested →
Failure planted on purpose
The bad samples were written, injected or swapped in on purpose, by a model, a script or by hand.
Dependency blindness
The agent writes its own version of something the codebase, the standard library or an installed package already provides. On 50 patches that re-implement something already available and 50 clean samples, with 5 ranked rows: No single best reviewer: 3 rows cannot be told apart at 95% on the samples they share (gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash); the highest J is +90.0% (95% interval +82.0% to +98.0%). No tool of the catalog ran a check of its own on this set. A shortcut baseline (No name from the shown file appears in the added lines) scores J +38.0% on the same set.
The clean side was selected on the outcome. 8 of 58 drafted pairs were dropped because the strong models flagged their clean side and a re-read found that the agent's own patch already re-implements something. Missed positives and unflagged clean sides were not re-read, so the step removed false alarms of exactly the models that are ranked. Bound: had the 8 clean sides stayed and been flagged, J would be about +77% for gpt-6-luna instead of +90%, and about +72% for deepseek-v4-flash and +72% for mimo.
J 80% or more on our set · Data: Hand-built on SWE-rebench-OpenHands-Trajectories (Nebius, CC-BY-4.0): correct agent patches, one call re-implemented by hand. What dependency blindness is → · All rows → · How it was tested →
Fabrication
Code that calls functions, methods or packages that don't exist. On 348 fabricated completions and 288 clean samples, with 15 ranked rows: No single best reviewer: 4 rows cannot be told apart at 95% on the samples they share (deepseek-v4.1-flash, glm-5.3, deepseek-v4-flash and gpt-6-luna); the highest J is +94.3% (95% interval +91.6% to +96.7%). The best catalog tool running its own check is pyright, at J +85.3%. A shortcut baseline (Context-only rule) scores J +59.7% on the same set.
J 80% or more on our set · Data: Delulu (microsoft/delulu), Python rows. What fabrication is → · All rows → · How it was tested →
Obsolescence
The code uses a deprecated or removed library API. On 155 solutions calling a deprecated API and 155 clean samples, with 9 ranked rows: No single best reviewer: 3 rows cannot be told apart at 95% on the samples they share (gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash); the highest J is +89.7% (95% interval +82.9% to +95.1%). The best catalog tool running its own check is pyright (latest libraries), at J +56.8%. A shortcut baseline (The code uses an API listed in the 2024 deprecation map) scores J +97.4% on the same set.
J 80% or more on our set · Data: Wang et al. (ICSE'25) deprecated-API map (65 pairs, 8 libraries) + solutions written by 5 models, deprecated call swapped in. What obsolescence is → · All rows → · How it was tested →
Scope creep
The agent makes changes nobody asked for while solving the task. On 175 patches with an injected unrequested change and 175 clean samples, with 5 ranked rows: Best reviewer: mimo-v2.6-flash at J +97.7%. Its paired difference from the next row (deepseek-v4-flash) has a 95% interval that excludes 0. No tool of the catalog ran a check of its own on this set. A shortcut baseline (Two or more hunks in files that already existed) scores J +46.9% on the same set.
The clean side was tuned after looking at the reviewers' answers. With clean patches matched on file count only (54 + 54), J was +57% for deepseek-v4-flash, +43% for gpt-6-luna and +28% for grok-4.1-fast (mimo +91%). The clean side was then limited to near-minimal patches, a size rule chosen after reading which clean patches the three strong models flagged: +84% / +83% / +58% on 64 + 64 (mimo +94%). The published set uses the filtered clean side.
J 80% or more on our set · Data: Synthetic, built on SWE-rebench-OpenHands-Trajectories (Nebius, CC-BY-4.0): correct agent patches + one change injected by Claude Opus. What scope creep is → · All rows → · How it was tested →
Supply chain
The agent adds a dependency that is a malicious lookalike of a real package. On 148 patches adding a known-malicious package and 148 clean samples, with 5 ranked rows: No single best reviewer: 2 rows cannot be told apart at 95% on the samples they share (gpt-6-luna and deepseek-v4-flash); the highest J is +77.0% (95% interval +70.3% to +83.8%). No tool of the catalog ran a check of its own on this set. A shortcut baseline (An added package is in the OpenSSF malicious-packages list) scores J +100.0% on the same set.
The set was tuned by looking at scores: difficulty and the mix of genuine names were set over three rounds (easy canary: deepseek-v4-flash +100%, a popularity rule +95%; hard canary of 20 pairs: gpt-6-luna +90%, the name rule +45%; published mix: the name rule +84%, gpt-6-luna +77%). The mix of genuine names decides which of the two is ahead.
J 25% to under 80% on our set · Data: Synthetic, hand-designed on SWE-rebench-OpenHands-Trajectories (Nebius, CC-BY-4.0) + OpenSSF malicious-packages names. What supply chain is → · All rows → · How it was tested →
Test edits that hide failures · failure mode: Test manipulation
The agent edits, weakens or skips tests so failures disappear instead of fixing the code. On 56 sessions where the agent edits tests to hide a failure and 249 clean samples, with 12 ranked rows: No single best reviewer: 8 rows cannot be told apart at 95% on the samples they share (minimax-m3, giskard, Qwen3.8-27B (self-hosted) and 5 more); the highest J is +91.0% (95% interval +81.3% to +94.9%). No tool of the catalog ran a check of its own on this set. A shortcut baseline (First user message mentions tests) scores J +60.3% on the same set.
J 80% or more on our set · Data: Patronus TRACE (PatronusAI/trace-dataset). What test manipulation is → · All rows → · How it was tested →
Not benchmarked yet
No public dataset with confirmed labels exists for cascading failure and context pollution, so nothing is scored for them. The leaderboard lists the catalog's tools for each by GitHub stars, which measure popularity, not detection.
- Cascading failure: The agent detects an error and attempts a fix that introduces NEW errors; repeated iterations spiral further from the working state. Catalog tools →
- Context pollution: The agent's working context degrades over a session — relevant prior information is dropped, irrelevant material accumulates, or hallucinations from earlier turns are treated as ground truth in later turns. Catalog tools →
Read before citing
- J compares reviewers within one mode. Do not compare J across modes or average it over them: each set was built differently (real agent runs, model-written functions, injected or hand-built failures) and has its own sample unit, and where the bad samples come from explains most of the difference between modes.
- Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product. In use nobody knows which failure a change has, so the prompt of every mode would run on every change and their false alarms add up; the any-flag estimate next to the grid shows by how much. No run with all prompts on the same changes has been made.
- Samples are grouped in 6 of 9 modes (Fabrication: 163 files; Security vulnerability: 166 tasks; Logic error: 379 tasks; Obsolescence: 69 API pairs; Supply chain: 148 pairs; Dependency blindness: 50 pairs). Intervals there resample whole groups (100,000 draws). A reviewer is named best only when its paired difference from the runner-up, on the samples both answered, has a 95% interval that excludes 0; otherwise the rows that cannot be told apart are listed as a top group. The best row is still a pick on the test set, so its J is a little optimistic.
- Contamination was not assessed. Every set draws on public material (problem sets, GitHub issues and their fixes, CVE fixes, library release notes, package lists), so a model may have seen a task or its answer. Sources: Delulu (Microsoft) (public since 2026-03); Patronus TRACE (public since 2026-01); SusVibes (LeiLiLab) (public since 2025-12); Collu-Bench (lt-asset) (public since 2024-10); SWE-rebench-OpenHands-Trajectories (Nebius) (public since 2025-09); Wang et al. deprecated-API map (ICSE'25) (public since 2024-06); OpenSSF malicious-packages (public since 2023-03). The reviewers' training cut-off dates are not recorded here.
- Rates are on answers that carry a verdict. 37 of 116 rows have unreadable answers (the count is beside each row, per label); 10 of them have fewer than 95% readable answers in one class and are marked incomplete.
- Same prompt for every model within a mode, temperature 0 (gpt-6-luna: its default), one run each; run-to-run variation was not measured. Prompts and tool settings were fixed before scoring. mypy, ruff and the simple rules are shown in the best of the configurations we ran in each mode, which is a pick on the test set; pyright and mypy on Obsolescence ran with the latest libraries installed.
- Baseline rules and shortcut baselines are shown for reference. They are never counted as best and never enter a finding. Rows outside a tool's design scope are listed apart and are not ranked.
- A row's cost covers every call made to get its scored answers: the current answer of each sample plus earlier attempts that failed or were cut off and had to be re-run. Smoke tests, runs on earlier versions of a set and accidental duplicate answers are not included. Cost per 1,000 reviews is that cost divided by the reviews made, for one mode-specific question per review. Checking a change for every failure costs the sum over the modes.
- The findings above are computed from the rows, so they change with them.
- A rule that needs no understanding of the failure reaches J 25% or more on 9 of 9 sets: Fabrication (+60%: context-only rule); Test edits that hide failures (+60%: first user message mentions tests); Security vulnerability (+49%: task prior); Logic error (+39%: surface classifier); False 'done' claims (+61%: issue prior); Scope creep (+47%: two or more hunks in files that already existed); Obsolescence (+97%: the code uses an API listed in the 2024 deprecation map); Supply chain (+100%: an added package is in the OpenSSF malicious-packages list); Dependency blindness (+38%: no name from the shown file appears in the added lines). High reviewer scores on those sets may partly reflect the same shortcut; J 0 is not the right zero there.
The full table of threats to validity →
How it was measured
Each sample is a task plus what an agent or a model produced for it (a patch, a function or a whole session), labelled bad if it contains the failure and clean if it does not. How the samples of each set were made is stated per set above and on the benchmark page. Every reviewer sees the samples without the labels.
- An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. j_strict is J under the old rule that counted such answers as 'not flagged'; j_bounds is J if every unreadable answer had been wrong, and right.
- 95% intervals. TPR and FPR: Wilson. J: Newcombe where samples are independent; where samples are grouped (task, pair, API pair, file) whole groups are resampled (100,000 draws, fixed seed).
- Two rows are compared on the samples both have a readable verdict for; the difference of their J gets a 95% bootstrap interval (groups resampled where the mode has them). 'Ahead', 'worse' and a single winner need an interval that excludes 0 at the published precision: a bound that prints as 0.0% does not exclude 0.
- Best reviewer: the best ranked row of a mode that is not marked incomplete. A row with under 95% readable answers in one class is listed and ranked, with its J bounds, but is never named best reviewer or sole winner; if its J is higher than the best reviewer's it stays in the top group. Not ranked: baseline rules, shortcut baselines and rows outside a tool's design scope. Top group: the best reviewer and every ranked row whose paired difference from it includes 0. Best catalog tool: the best tool of the SAICA catalog that ran its own check; a framework that ran our rule text is not a candidate.
- Tiers describe the best reviewer's J on our set, not how well a failure is caught in use: J 80% or more on our set; J 25% to under 80% on our set; J under 25% on our set.
- A row's cost covers every call made to get its scored answers: the current answer of each sample plus earlier attempts that failed or were cut off and had to be re-run. Smoke tests, runs on earlier versions of a set and accidental duplicate answers are not included. Cost per 1,000 reviews is that cost divided by the reviews made, for one mode-specific question per review. 'Cheap' means at most $5.00 per 1,000 reviews.
Source datasets: SWE-rebench-OpenHands-Trajectories (Nebius), SusVibes (LeiLiLab), Collu-Bench (lt-asset), Delulu (Microsoft), Wang et al. deprecated-API map (ICSE'25), OpenSSF malicious-packages and Patronus TRACE. The full setup and caveats of each set are in the benchmark methodology; the raw numbers are in results.json (CC-BY-4.0).
Cite
Vasyl Rakivnenko. Can supervision tools and LLMs catch AI coding agent failures? Findings from SAICA-Bench. Stanford University, 2026. SAICA-Bench results v2.0.0. https://saica.ai/findings/
Leaderboard → · All failure modes → · Benchmark methodology → · Build a supervision kit →