AI coding agent says “done” but isn’t: incomplete execution
Incomplete execution, also called premature or false completion, is when an AI coding agent reports a task as done while parts of it are skipped, stubbed out or silently dropped. Common forms are TODO comments left in the code, function bodies replaced with pass or ..., a feature only partly implemented, error paths skipped, or the agent stopping before every item the user asked for is addressed. It is the opposite of scope creep: the agent under-delivers inside the request instead of reaching beyond it.
A false “done” hands the checking back to the person who delegated the task. In Answer.AI’s month-long evaluation of Devin, 3 of 20 tasks were completed.
At a glance: On SAICA-Bench: No reviewer reaches J 25% on this set: the highest J of a row with enough readable answers is +19.5% (95% interval +13.9% to +24.9%), and 3 other rows cannot be told apart from it. No best reviewer is named. Results ↓
- premature completion
- false completion
- claimed done
- partial completion
- stub left behind
- false-done
- task abandonment
- declared done not done
- early termination
- unfinished task
- partial completion claimed complete
How well can false “done” claims be caught?
J under 25% on our set SAICA-Bench scores this mode as false 'done' claims. The agent declares the task done, but the issue's hidden tests fail. That covers wrong fixes as well as unfinished work. No reviewer reaches J 25% on this set: the highest J of a row with enough readable answers is +19.5% (95% interval +13.9% to +24.9%), and 3 other rows cannot be told apart from it. No best reviewer is named.
Best reviewer
any ranked row; a tie is shown as a top group
no reviewer reaches J 25%
Highest J +19.5% (95% interval +13.9% to +24.9%)
Best catalog tool
a tool of the catalog running its own check
None reaches J 25% on this set, so none is named.
Shortcut baseline
a rule that understands nothing about the failure
Issue prior
J +61.3%. Read the reviewers against this, not against 0.
What was tested
- Bad samples
- Real agent output. Qwen3-Coder-480B in OpenHands, working on real GitHub issues (SWE-rebench). Label: execution: the issue's hidden tests fail after the agent declared the task done. Clean samples: runs of the same agent on other issues where the hidden tests pass.
- One sample
- one agent run (issue, final patch, closing message); one run per issue.
- Samples scored
- 600 runs declared done whose hidden tests fail and 600 clean (50% of the set is bad).
- Measure
- J under 25% on our set.
- Shortcut baseline
- A rule with no understanding of the failure scores J +61.3% here (Issue prior). Read reviewer scores against it, not against 0.
All results on this set
Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.
| # | Reviewer | 95% interval | bad samples flagged | clean samples flagged | No verdictanswers left out | False alarmsper 100 clean changes | Precisionif 1% of changes are bad | Precisionif 5% of changes are bad | Precisionif 20% of changes are bad | per 1,000 reviews |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 |
deepseek-v4-flash, asked directly
Run card
|
+20.3%+14.3% to +26.1% | 72.7%68.6% to 76.5% | 52.4%47.9% to 56.8% | 230 of 1,200 (incomplete) | 52.4 | 1% | 7% | 26% | $4.95api |
| 2 |
mimo-v2.6-flash, asked directly
Run card
|
+20.0%+14.5% to +25.2% | 43.0%38.9% to 47.1% | 23.0%19.7% to 26.7% | 81 of 1,200 (incomplete) | 23 | 2% | 9% | 32% | $4.36api |
| 3 |
gpt-6-luna, asked directly
Run card
|
+19.5%+13.9% to +24.9% | 59.7%55.7% to 63.5% | 40.2%36.3% to 44.1% | 0 | 40.2 | 1% | 7% | 27% | $1.05list |
| 4 |
PR-Agent review, any focus area listed (deepseek-v4-flash)
Run card
How this run differs from the tool's defaults
|
+14.7%+3.4% to +25.5% | 67.8%59.8% to 74.9% | 53.1%45.0% to 61.1% | 14 of 300 | 53.1 | 1% | 6% | 24% | ~$15.36api |
| 5 |
Arize Phoenix CompletenessEvaluator (deepseek-v4-flash)
Run card
How this run differs from the tool's defaults
|
+6.6%+3.2% to +10.0% | 13.1%10.7% to 16.1% | 6.5%4.8% to 8.8% | 10 of 1,200 | 6.5 | 2% | 10% | 33% | $0.82api |
| 6 |
Opik AgentTaskCompletionJudge, score < 0.5 (deepseek-v4-flash)
Run card
How this run differs from the tool's defaults
|
+3.6%−3.3% to +10.6% | 11.6%7.3% to 17.7% | 8.0%4.6% to 13.5% | 3 of 300 | 8 | 1% | 7% | 27% | ~$2.15api |
| 7 |
grok-4.1-fast (no reasoning), asked directly
Run card
|
+2.5%−0.1% to +5.2% | 95.7%93.7% to 97.0% | 93.2%90.9% to 94.9% | 0 | 93.2 | 1% | 5% | 20% | $0.75list |
| 8 |
DeepEval TaskCompletionMetric (deepseek-v4-flash)
Run card
How this run differs from the tool's defaults
|
+0.8%−0.6% to +2.4% | 1.9%1.0% to 3.3% | 1.0%0.5% to 2.2% | 10 of 1,200 | 1 | 2% | 9% | 32% | $0.46api |
| 9 |
ruff FIX rules (TODO / FIXME / XXX / HACK comments)
Run card
How this run differs from the tool's defaults
|
+0.5%−0.2% to +1.5% | 0.5%0.2% to 1.5% | 0.0%0.0% to 0.6% | 0 | 0 | 100% | 100% | 100% | $0free |
| 10 |
Phi-4-mini, asked directly
Run card
|
0.0%−1.0% to +1.0% | 0.5%0.2% to 1.5% | 0.5%0.2% to 1.5% | 14 of 1,200 | 0.5 | 1% | 5% | 20% | $0.27list |
Shown for reference, not ranked
These rows are never counted as best. A shortcut baseline shows what a rule with no understanding of the failure scores on this set; read the reviewers' scores against it.
| Row and why it is not ranked | J95% interval | TPR | FPR | No verdict | Costper 1,000 reviews |
|---|---|---|---|---|---|
|
Stub rule: TODO / FIXME / pass / ... / NotImplementedError added to a source file
Not ranked: baseline rule: shown for reference, never counted as best.
Run card
|
+2.5%−0.7% to +5.7% | 10.0% | 7.5% | 0 | $0 |
|
Issue prior: the agent's other runs of this issue mostly fail
Not ranked: shortcut baseline: shows what a rule with no understanding of the failure scores on this set.
Flags a run when fewer than half of the other submitted runs of the same issue pass the hidden tests (or when there is no other run). It reads neither the patch nor the closing message.
|
+61.3%+56.6% to +65.5% | 79.5% | 18.2% | 0 | $0 |
How to read these numbers
- J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.
- An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. j_strict is J under the old rule that counted such answers as 'not flagged'; j_bounds is J if every unreadable answer had been wrong, and right.
- 95% intervals. TPR and FPR: Wilson. J: Newcombe where samples are independent; where samples are grouped (task, pair, API pair, file) whole groups are resampled (100,000 draws, fixed seed).
- Best reviewer: the best ranked row of a mode that is not marked incomplete. A row with under 95% readable answers in one class is listed and ranked, with its J bounds, but is never named best reviewer or sole winner; if its J is higher than the best reviewer's it stays in the top group. Not ranked: baseline rules, shortcut baselines and rows outside a tool's design scope. Top group: the best reviewer and every ranked row whose paired difference from it includes 0. Best catalog tool: the best tool of the SAICA catalog that ran its own check; a framework that ran our rule text is not a candidate.
- A row's cost covers every call made to get its scored answers: the current answer of each sample plus earlier attempts that failed or were cut off and had to be re-run. Smoke tests, runs on earlier versions of a set and accidental duplicate answers are not included. Cost per 1,000 reviews is that cost divided by the reviews made, for one mode-specific question per review. 'Cheap' means at most $5.00 per 1,000 reviews.
- Precision is the share of flagged changes that are really bad, if the stated share of all changes were bad. It is computed from the row's TPR and FPR; the set's own share of bad samples is 50%.
The same answers, scored another way
Parts of the set, or the same set under another labelling. They show how far the headline J depends on how the set was built.
Pure issues · 381 bad, 381 clean
Failing runs of issues the agent never solves in any other run, against passing runs of issues it always solves. Here the issue alone gives the label away.
| Row | J | 95% interval | TPR | FPR |
|---|---|---|---|---|
| deepseek-v4-flash | +26.9% | +19.2% to +34.2% | 71.0% | 44.1% |
| mimo-v2.6-flash | +25.0% | +18.3% to +31.4% | 43.0% | 18.0% |
| gpt-6-luna | +24.7% | +17.7% to +31.3% | 59.1% | 34.4% |
| PR-Agent | +21.2% | +6.0% to +35.2% | 63.3% | 42.1% |
| Arize Phoenix | +8.6% | +4.3% to +13.0% | 14.7% | 6.1% |
| grok-4.1-fast | +3.7% | −0.2% to +7.6% | 94.0% | 90.3% |
| Opik | +3.4% | −5.5% to +11.8% | 9.7% | 6.3% |
| stub rule | +1.6% | −2.0% to +5.2% | 7.3% | 5.8% |
| DeepEval | +1.3% | −0.5% to +3.4% | 2.1% | 0.8% |
| ruff FIX rules | +0.5% | −0.5% to +1.9% | 0.5% | 0.0% |
| Phi-4-mini | 0.0% | −1.5% to +1.4% | 0.5% | 0.5% |
Mixed issues · 208 bad, 214 clean
Runs of issues that the agent solves in some runs and not in others. The issue no longer gives the label away, so this is the closer test of telling a finished run from an unfinished one.
| Row | J | 95% interval | TPR | FPR |
|---|---|---|---|---|
| mimo-v2.6-flash | +11.3% | +1.8% to +20.5% | 42.5% | 31.2% |
| PR-Agent | +10.3% | −6.6% to +25.7% | 76.0% | 65.7% |
| gpt-6-luna | +9.6% | +0.1% to +18.9% | 60.1% | 50.5% |
| deepseek-v4-flash | +8.6% | −1.0% to +18.0% | 75.3% | 66.7% |
| stub rule | +4.2% | −2.3% to +10.6% | 14.9% | 10.8% |
| Opik | +3.9% | −7.6% to +16.9% | 13.7% | 9.9% |
| Arize Phoenix | +3.0% | −2.4% to +8.6% | 10.1% | 7.1% |
| ruff FIX rules | +0.5% | −1.3% to +2.7% | 0.5% | 0.0% |
| grok-4.1-fast | +0.4% | −2.5% to +3.4% | 98.6% | 98.1% |
| Phi-4-mini | 0.0% | −2.2% to +2.3% | 0.5% | 0.5% |
| DeepEval | −0.4% | −3.2% to +2.2% | 1.0% | 1.4% |
More tables for this set in results.json
issue_structure: Each sample is one run. The other submitted runs of the same issue say how hard the issue is for this agent.closing_messageproposed_naming
They are under modes[].tables of results.json.
How it was tested
- 600 runs where the agent finished and declared the task done but the issue's hidden tests fail, vs 600 where it declared done and the tests pass. One run per issue; an issue is never on both sides.
- Reviewers see the issue, the agent's final patch and its closing message, and no tests. Same prompt for every model; temperature 0 (gpt-6-luna: its default). Run 2026-11-20.
Caveats
- The KG's stub signal (TODO, pass, NotImplementedError in the diff) does not define the label: across 60,739 finished runs, stub markers appear about as often in passing runs (2,092) as in failing ones (2,208).
- All runs come from one agent (Qwen3-Coder-480B with OpenHands).
- The KG's task-completion judges (Phoenix, Opik, DeepEval; each through its own built-in check, with deepseek-v4-flash as its model) rarely call a finished run incomplete (they flag 2% to 13% of failing runs): given the agent's own 'done' message, they mostly believe it. PR-Agent's review lists a focus area for 68% of failing runs and for 53% of passing ones. Not runnable on a finished patch: PR-Agent ticket compliance (needs a linked GitHub or Jira ticket), TruLens Plan Adherence and DeepEval Step Efficiency (need the agent's step traces).
- Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.
- Answers with no verdict are left out of the rates; they are not counted as 'not flagged'. Unreadable: deepseek-v4-flash 230 of 1,200 (113 bad, 117 clean; mostly answers cut by the token budget); mimo-v2.6-flash 81 of 1,200 (46 bad, 35 clean; mostly answers cut by the token budget); Phi-4-mini 14 of 1,200 (7 bad, 7 clean; mostly provider or connection errors); PR-Agent 14 of 300 (7 bad, 7 clean; mostly the tool failed to run); Arize Phoenix 10 of 1,200 (6 bad, 4 clean; mostly requests the provider blocked); DeepEval 10 of 1,200 (6 bad, 4 clean; mostly requests the provider blocked); Opik 3 of 300 (3 bad, 0 clean; mostly requests the provider blocked). Marked incomplete (under 95% readable in one class): deepseek-v4-flash, mimo-v2.6-flash.
- Run on part of the samples: Opik (150 bad + 150 clean); PR-Agent (150 bad + 150 clean). Their rates are on those samples, and comparisons with other rows use only the samples both ran on.
- What is measured: the agent declared the task done and the issue's hidden tests fail. That covers wrong fixes as well as unfinished work, so the result is about false 'done' claims, not about the knowledge graph's incomplete execution (skipped or stubbed subtasks).
- The label is mostly decided by the issue, not by the run: 64% of the failing runs come from issues the agent solves in none of its other runs, and 64% of the passing runs from issues it solves in all of them. A rule that never sees the run (do the agent's other runs of this issue mostly fail?) scores J +61%; the highest reviewer J is +20%.
- On issues with both outcomes (208 failing, 214 passing runs) the three highest-scoring models reach J +9% to +11%, each with an interval of about +/-9 points; on the pure issues they reach +25% to +27%.
- The closing message carries no signal: it says the tests pass in 75% of the failing runs and 78% of the passing ones.
- The issues and their hidden tests are mined automatically (SWE-rebench) and were not checked by people, so some failing runs may fail because the tests ask for more than the issue text says.
False 'done' claims on the leaderboard → How this set was built → Threats to validity → Build a supervision kit →
How to detect it
Signals that point to incomplete execution, from the SAICA knowledge graph:
- Grep for
TODO,FIXME,XXX,NotImplementedError, or barepass/...in the generated diff. - Tests for features the user explicitly named don't exist.
- Agent's final message claims a subtask is done but the file contains only a stub.
- Function docstrings describe a behavior not implemented by the body.
Studies and reported patterns
Records in the catalog that are not single dated incidents: findings of a study, recurring behaviour that was reported, or scenarios a source describes as hypothetical.
-
Devin spends >24h hallucinating Railway features rather than recognizing platform limit
Study finding · 2025-01-08
Answer.AI's month-long Devin eval: 3 of 20 tasks completed. In one, Devin spent a day pursuing a Railway single-deployment feature that does not exist, fabricating plausible steps.
Tools in the catalog that address it
7 tools are tagged with incomplete execution in the catalog, most-starred first. A tag is the catalog's claim about what a tool is for; it is not a test result. Where SAICA-Bench ran the tool on this mode, the measured J is shown: that row is the evidence.
- Ruff measured J +0.5% Extremely fast Python linter and formatter, written in Rust.
- ESLint Pluggable JavaScript/TypeScript linter and code-quality enforcer.
- opik measured J +3.6% Debug, evaluate, and monitor LLM applications, RAG systems, and agentic workflows.
- DeepEval measured J +0.8% The open-source LLM evaluation framework for testing and evaluating LLM applications across 20+ custom and researched metrics.
- pr-agent measured J +14.7% AI-powered tool for automated pull-request analysis, feedback, and suggestions.
- Arize Phoenix measured J +6.6% Open-source LLM tracing and evaluation platform built on OpenTelemetry.
- TruLens Evaluation and Tracking for LLM Experiments and AI Agents
In other taxonomies
How incomplete execution maps to other failure taxonomies. “Exact” means the same failure, “partial” an overlap, “related” a neighbouring category.
-
MAST: Multi-Agent System Failure Taxonomy
Mapping drafted by an AI model from the source text; not yet reviewed by a person.
-
partial
FM-3.1Premature termination — Ending a task before its objectives are met overlaps with incomplete_execution (stopping before all user-specified items are addressed). Not exact: the SAICA mode is defined by a false "done" claim, which MAST does not require, and FM-3.1 also covers early stops with no such claim. Was "exact" before the fix round of 2026-11-21.
Related:
FM-3.2No or incomplete verification -
partial
-
9 Critical Failure Patterns of Coding Agents
Mapping drafted by an AI model from the source text; not yet reviewed by a person.
Related:
DAP-09Exception & Error Handling -
Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes
Mapping drafted by an AI model from the source text; not yet reviewed by a person.
Related:
SHAH-CAT-02Agent Lifecycle and State;SHAH-CAT-12Failure Handling & Implementation Robustness
Related failure modes
- Scope creep in AI coding agents — Changes nobody asked for: other files, new packages, config edits, unrequested refactors.
- Logic errors in AI-generated code — Real APIs used correctly, but the code computes the wrong result.
Research
Questions and answers
What is incomplete execution in AI coding agents?
Incomplete execution, also called premature or false completion, is when an AI coding agent reports a task as done while parts of it are skipped, stubbed out or silently dropped. Common forms are TODO comments left in the code, function bodies replaced with pass or ..., a feature only partly implemented, error paths skipped, or the agent stopping before every item the user asked for is addressed. It is the opposite of scope creep: the agent under-delivers inside the request instead of reaching beyond it.
How do you detect false “done” claims?
Signals that point to it: Grep for TODO, FIXME, XXX, NotImplementedError, or bare pass/... in the generated diff. Tests for features the user explicitly named don't exist. Agent's final message claims a subtask is done but the file contains only a stub. Function docstrings describe a behavior not implemented by the body. On SAICA-Bench: No reviewer reaches J 25% on this set: the highest J of a row with enough readable answers is +19.5% (95% interval +13.9% to +24.9%), and 3 other rows cannot be told apart from it. No best reviewer is named.
Which reviewers catch false “done” claims on SAICA-Bench, and how well?
No reviewer reaches J 25% on this set: the highest J of a row with enough readable answers is +19.5% (95% interval +13.9% to +24.9%), and 3 other rows cannot be told apart from it. No best reviewer is named. A rule with no understanding of the failure scores J +61.3% on the same set (Issue prior), so read the scores against that, not against 0. Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.
How was detection of false “done” claims measured?
On 1,200 scored samples: 600 runs declared done whose hidden tests fail and 600 clean ones. Where the bad samples come from: real agent output (Qwen3-Coder-480B in OpenHands, working on real GitHub issues (SWE-rebench)). Label: execution: the issue's hidden tests fail after the agent declared the task done. Data: SWE-rebench-OpenHands-Trajectories (Nebius, CC-BY-4.0): Qwen3-Coder-480B in OpenHands on real GitHub issues. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.
Benchmark results from 2026-11-22. The definition, detection signals and tags come from the SAICA knowledge graph.