Logic errors in AI-generated code
A logic error is when an AI coding agent uses real, current APIs correctly but writes logic that computes the wrong result for the task, for example an inverted condition, an off-by-one or a faulty algorithm. It is the classic “passes the type check, fails correctness” bug. It differs from code hallucination, where the API does not exist, and from obsolescence, where the API is out of date. A fix that makes things worse is a cascading failure.
Nothing about the code looks wrong to a compiler, type checker or linter, so only tests, execution or a reader who understands the task can catch it. In the Gemini CLI case in the catalog, the agent misread a failed command as success and went on to destroy the user’s files.
At a glance: On SAICA-Bench: Best reviewer: mimo-v2.6-flash at J +74.3%. Its paired difference from the next row (deepseek-v4-flash) has a 95% interval that excludes 0. Results ↓
- semantic bug
- off-by-one
- control-flow bug
- incorrect implementation
- faulty algorithm
- reasoning error
- inverted condition
How well can logic errors in AI-generated code be caught?
J 25% to under 80% on our set The code runs but computes the wrong result. Best reviewer: mimo-v2.6-flash at J +74.3%. Its paired difference from the next row (deepseek-v4-flash) has a 95% interval that excludes 0.
Best reviewer
any ranked row; a tie is shown as a top group
mimo-v2.6-flash
Highest J +74.3% (95% interval +69.4% to +78.9%)
Best catalog tool
a tool of the catalog running its own check
J +53.2% (95% interval +47.9% to +58.3%)
Shortcut baseline
a rule that understands nothing about the failure
Surface classifier
J +39.4%. Read the reviewers against this, not against 0.
What was tested
- Bad samples
- Model-written code, not agent output. 11 code models of 2023-24 (CodeLlama, DeepSeek-Coder, StarCoder2, Llama 3, gpt-4o-mini) answering MBPP and HumanEval tasks; not agent runs. Label: execution: the function fails a test assertion of its task. Clean samples: the reference solutions of the same tasks.
- One sample
- one function; functions of the same task are grouped (intervals resample tasks).
- Samples scored
- 1,536 wrong functions and 904 clean (63% of the set is bad).
- Measure
- J 25% to under 80% on our set.
- Shortcut baseline
- A rule with no understanding of the failure scores J +39.4% here (Surface classifier). Read reviewer scores against it, not against 0.
All results on this set
Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.
| # | Reviewer | 95% interval | bad samples flagged | clean samples flagged | No verdictanswers left out | False alarmsper 100 clean changes | Precisionif 1% of changes are bad | Precisionif 5% of changes are bad | Precisionif 20% of changes are bad | per 1,000 reviews |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 |
mimo-v2.6-flash, asked directly
Run card
|
+74.3%+69.4% to +78.9% | 89.0%87.3% to 90.5% | 14.7%12.5% to 17.1% | 4 of 2,440 | 14.7 | 6% | 24% | 60% | $0.23api |
| 2 |
deepseek-v4-flash, asked directly
Run card
|
+69.1%+64.1% to +73.8% | 90.8%89.2% to 92.1% | 21.7%19.1% to 24.5% | 8 of 2,440 | 21.7 | 4% | 18% | 51% | $0.31api |
| 3 |
gpt-6-luna, asked directly
Run card
|
+68.5%+63.1% to +73.5% | 89.3%87.6% to 90.7% | 20.8%18.3% to 23.6% | 0 | 20.8 | 4% | 18% | 52% | $0.15list |
| 4 |
PR-Agent review, any focus area listed (deepseek-v4-flash)
Run card
How this run differs from the tool's defaults
|
+53.2%+47.9% to +58.3% | 91.4%89.9% to 92.7% | 38.2%35.1% to 41.4% | 1 of 2,440 | 38.2 | 2% | 11% | 37% | $1.29api |
| 5 |
uqlm CodeGenUQ: 5 fresh solutions, functional-equivalence rate < 0.5 (deepseek-v4-flash)
Run card
How this run differs from the tool's defaults
|
+52.5%+44.4% to +60.1% | 96.0%93.1% to 97.7% | 43.5%37.8% to 49.4% | 23 of 600 (incomplete) | 43.5 | 2% | 10% | 36% | ~$3.81api |
| 6 |
Phi-4-mini, asked directly
Run card
|
+22.0%+17.2% to +26.7% | 61.7%59.2% to 64.1% | 39.7%36.5% to 43.0% | 117 of 2,440 (incomplete) | 39.7 | 2% | 8% | 28% | $0.07list |
| 7 |
grok-4.1-fast (no reasoning), asked directly
Run card
|
+11.9%+8.8% to +15.0% | 93.5%92.1% to 94.6% | 81.6%79.0% to 84.0% | 0 | 81.6 | 1% | 6% | 22% | $0.14list |
Shown for reference, not ranked
These rows are never counted as best. A shortcut baseline shows what a rule with no understanding of the failure scores on this set; read the reviewers' scores against it.
| Row and why it is not ranked | J95% interval | TPR | FPR | No verdict | Costper 1,000 reviews |
|---|---|---|---|---|---|
|
pyright
Not ranked: outside the tool's design scope: a type checker looks for type errors, not for wrong results; shown as a floor.
Run card
How this run differs from the tool's defaults
|
+0.3%−0.5% to +1.1% | 1.5% | 1.2% | 0 | $0 |
|
mypy (--check-untyped-defs --ignore-missing-imports)
Not ranked: outside the tool's design scope: a type checker looks for type errors, not for wrong results; shown as a floor.
Run card
How this run differs from the tool's defaults
|
−0.7%−2.0% to +0.5% | 2.6% | 3.3% | 0 | $0 |
|
ruff (pyflakes + bugbear rules)
Not ranked: outside the tool's design scope: a linter looks for lint findings, not for wrong results; shown as a floor.
Run card
How this run differs from the tool's defaults
|
−3.1%−5.4% to −0.9% | 4.3% | 7.4% | 0 | $0 |
|
Surface classifier: code tokens only, task never seen
Not ranked: shortcut baseline: shows what a rule with no understanding of the failure scores on this set.
A logistic regression on which code tokens occur in the function, trained on four fifths of the tasks and applied to the remaining fifth. It never reads the task and runs nothing.
|
+39.4%+34.6% to +44.1% | 69.7% | 30.3% | 0 | $0 |
|
Formatting cues: any of four layout habits
Not ranked: shortcut baseline: shows what a rule with no understanding of the failure scores on this set.
Flags a function that has a single-quoted string, trailing whitespace, a blank line inside the body or 'return (...)'. Correct functions are tidy reference solutions; wrong ones are raw model output.
|
+34.6%+30.1% to +39.1% | 59.8% | 25.2% | 0 | $0 |
How to read these numbers
- J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.
- An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. j_strict is J under the old rule that counted such answers as 'not flagged'; j_bounds is J if every unreadable answer had been wrong, and right.
- 95% intervals. TPR and FPR: Wilson. J: Newcombe where samples are independent; where samples are grouped (task, pair, API pair, file) whole groups are resampled (100,000 draws, fixed seed).
- Best reviewer: the best ranked row of a mode that is not marked incomplete. A row with under 95% readable answers in one class is listed and ranked, with its J bounds, but is never named best reviewer or sole winner; if its J is higher than the best reviewer's it stays in the top group. Not ranked: baseline rules, shortcut baselines and rows outside a tool's design scope. Top group: the best reviewer and every ranked row whose paired difference from it includes 0. Best catalog tool: the best tool of the SAICA catalog that ran its own check; a framework that ran our rule text is not a candidate.
- A row's cost covers every call made to get its scored answers: the current answer of each sample plus earlier attempts that failed or were cut off and had to be re-run. Smoke tests, runs on earlier versions of a set and accidental duplicate answers are not included. Cost per 1,000 reviews is that cost divided by the reviews made, for one mode-specific question per review. 'Cheap' means at most $5.00 per 1,000 reviews.
- Precision is the share of flagged changes that are really bad, if the stated share of all changes were bad. It is computed from the row's TPR and FPR; the set's own share of bad samples is 63%.
The same answers, scored another way
Parts of the set, or the same set under another labelling. They show how far the headline J depends on how the set was built.
Mbpp · 882 bad, 477 clean
MBPP tasks only: one-line task descriptions, three hidden asserts per task.
| Row | J | 95% interval | TPR | FPR |
|---|---|---|---|---|
| mimo-v2.6-flash | +57.8% | +50.2% to +65.0% | 81.6% | 23.8% |
| gpt-6-luna | +51.7% | +44.0% to +59.1% | 82.1% | 30.4% |
| deepseek-v4-flash | +51.2% | +43.8% to +58.4% | 84.4% | 33.1% |
| PR-Agent | +37.9% | +31.0% to +44.6% | 85.3% | 47.4% |
| uqlm | +36.6% | +25.4% to +47.4% | 92.4% | 55.8% |
| Phi-4-mini | +20.4% | +13.9% to +26.9% | 64.5% | 44.2% |
| grok-4.1-fast | +11.3% | +7.1% to +15.6% | 91.4% | 80.1% |
| pyright | +0.6% | −0.3% to +1.6% | 0.8% | 0.2% |
| mypy | 0.0% | −1.5% to +1.6% | 1.7% | 1.7% |
| ruff | −4.5% | −7.8% to −1.2% | 5.0% | 9.4% |
Humaneval · 654 bad, 427 clean
HumanEval tasks only: the function signature and docstring are the task.
| Row | J | 95% interval | TPR | FPR |
|---|---|---|---|---|
| mimo-v2.6-flash | +94.5% | +91.3% to +97.2% | 98.9% | 4.5% |
| deepseek-v4-flash | +90.6% | +86.7% to +94.1% | 99.4% | 8.8% |
| gpt-6-luna | +88.9% | +83.6% to +93.4% | 98.9% | 10.1% |
| uqlm | +71.8% | +62.1% to +80.5% | 100.0% | 28.2% |
| PR-Agent | +71.8% | +65.0% to +78.4% | 99.7% | 27.9% |
| Phi-4-mini | +23.2% | +16.0% to +30.3% | 57.5% | 34.3% |
| grok-4.1-fast | +13.0% | +8.4% to +17.7% | 96.3% | 83.4% |
| pyright | +0.1% | −1.3% to +1.4% | 2.5% | 2.3% |
| mypy | −1.3% | −3.4% to +0.5% | 3.8% | 5.1% |
| ruff | −1.8% | −4.7% to +1.1% | 3.4% | 5.1% |
Gpt4o mini · 98 bad, 198 clean
Wrong functions written by gpt-4o-mini, the one recent generator in the set, against the correct reference solutions of the same tasks.
| Row | J | 95% interval | TPR | FPR |
|---|---|---|---|---|
| mimo-v2.6-flash | +33.2% | +18.7% to +46.9% | 60.2% | 27.0% |
| gpt-6-luna | +30.9% | +16.9% to +44.4% | 63.3% | 32.3% |
| deepseek-v4-flash | +27.4% | +13.4% to +40.7% | 62.2% | 34.9% |
| PR-Agent | +27.0% | +14.2% to +39.3% | 69.4% | 42.4% |
| pyright | −1.0% | −2.7% to 0.0% | 0.0% | 1.0% |
| mypy | −2.5% | −6.6% to +0.5% | 2.0% | 4.5% |
| ruff | −4.5% | −10.7% to +1.2% | 4.1% | 8.6% |
| Phi-4-mini | −6.1% | −17.9% to +5.5% | 41.2% | 47.3% |
| grok-4.1-fast | −11.3% | −18.5% to −4.5% | 79.6% | 90.9% |
More tables for this set in results.json
generatorstpr_by_generator: Share of each generator's wrong functions a reviewer flags, on readable answers. The false-positive rate is shared: every negative is a reference solution.formatting_cuessurface_classifierconsensus_disagreements: Rows where the three highest-scoring models all give the answer opposite to the label, among the rows all three answered readably. Labels follow the hidden asserts; reviewers were told to judge against the task text only.
They are under modes[].tables of results.json.
How it was tested
- 1536 functions that run but fail a test assertion vs 904 correct solutions, across 379 tasks. Code by 11 models (CodeLlama, DeepSeek-Coder, StarCoder2, Llama 3, gpt-4o-mini).
- Every label re-checked by running the code against its tests (the original asserts of MBPP and HumanEval; MBPP has three per task): crashes, timeouts, 'wrong' outputs that actually pass and reference solutions that fail were dropped.
- Reviewers see the task description and the function, not the tests. Same prompt for every model; temperature 0 (gpt-6-luna: its default). Run 2026-11-20.
Caveats
- Short, self-contained functions from two famous public problem sets: models may know the right answers, so expect lower scores on real agent code.
- Type checkers and linters look for a different kind of bug; they are here as a floor, not as fair competitors.
- KG tools run through their own built-in check, with deepseek-v4-flash as their model: uqlm re-solves the task 5 times and flags code that disagrees with most re-solves; it was run on the first 300 + 300 functions only (it makes 10 model calls per review).
- Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.
- Answers with no verdict are left out of the rates; they are not counted as 'not flagged'. Unreadable: Phi-4-mini 117 of 2,440 (69 bad, 48 clean; mostly answers without a verdict); uqlm 23 of 600 (1 bad, 22 clean; mostly no score returned); deepseek-v4-flash 8 of 2,440 (0 bad, 8 clean; mostly answers cut by the token budget); mimo-v2.6-flash 4 of 2,440 (1 bad, 3 clean; mostly answers cut by the token budget); PR-Agent 1 of 2,440 (0 bad, 1 clean; mostly the tool failed to run). Marked incomplete (under 95% readable in one class): Phi-4-mini, uqlm.
- Run on part of the samples: uqlm (300 bad + 300 clean). Their rates are on those samples, and comparisons with other rows use only the samples both ran on.
- The headline J blends two problem sets. For the three highest-scoring models, J is +89% to +94% on HumanEval and +51% to +58% on MBPP.
- 65% of the wrong functions were written by open code models of 8 billion parameters or fewer (2023-24). On the 98 wrong functions of the one recent generator, gpt-4o-mini, the same three models catch 60% to 63% (other generators: 82% to 97%), and their J against the reference solutions of the same tasks is +27% to +33%. The surface classifier (code tokens only) scores +30% on those rows.
- Every correct function is a reference solution and every wrong one is model-written, so layout gives the label away: a classifier that sees only code tokens, never the task, scores J +39%, and four formatting cues score +35%.
- Labels follow the hidden asserts, while reviewers are told to judge against the task text only. On MBPP, 111 of 882 'wrong' functions are flagged by none of the three highest-scoring models and 85 of 471 'correct' ones by all three; a hand check of 24 of these rows in the review of 21 November 2026 found the label did not hold under that question in 19.
- Textbook functions written by pre-agent models: this mode should not be averaged with the modes that score agent patches.
Logic error on the leaderboard → How this set was built → Threats to validity → Build a supervision kit →
How to detect it
Signals that point to logic error, from the SAICA knowledge graph:
- Unit tests fail with correctness assertions (not import or type errors).
- Execution in a sandbox produces outputs that contradict the task spec.
- Mutation testing reveals that substitution of logic primitives does not change test outcomes (test insensitivity is a different signal, but logic-error cases often co-occur).
- Symbolic or property-based tests find counterexamples.
Real-world incidents
-
Gemini CLI destroys a user's files after treating a failed mkdir as a success
2025-07-21 · data loss
Google's Gemini CLI took a failed mkdir for a success, then ran move commands that overwrote every file but one. The agent then told the user it had lost the data.
Tools in the catalog that address it
6 tools are tagged with logic error in the catalog, most-starred first. A tag is the catalog's claim about what a tool is for; it is not a test result. Where SAICA-Bench ran the tool on this mode, the measured J is shown: that row is the evidence.
- Ruff measured J −3.1% (not ranked: outside its design scope) Extremely fast Python linter and formatter, written in Rust.
- ESLint Pluggable JavaScript/TypeScript linter and code-quality enforcer.
- pr-agent measured J +53.2% AI-powered tool for automated pull-request analysis, feedback, and suggestions.
- Pyright measured J +0.3% (not ranked: outside its design scope) Microsoft's fast static type checker for Python.
- pytest Python test framework — discipline gate for "done means tests pass."
- uqlm measured J +52.5% Uncertainty Quantification for Language Models.
In other taxonomies
How logic error maps to other failure taxonomies. “Exact” means the same failure, “partial” an overlap, “related” a neighbouring category.
-
9 Critical Failure Patterns of Coding Agents
Mapping drafted by an AI model from the source text; not yet reviewed by a person.
-
partial
DAP-03Business Logic Mismatch — The post's wording, "code that runs without errors, but it produces the wrong output because the underlying logic is flawed", is the definition of logic_error. DAP-03 is the narrower side: it covers business rules, permissions and calculations only.
-
partial
-
Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes
Mapping drafted by an AI model from the source text; not yet reviewed by a person.
-
partial
SHAH-CAT-04Input Interpretation — Code that runs on real APIs but parses, validates or converts its input wrongly is a logic_error when a coding agent writes it. Narrower than logic_error (input handling only) and observed in framework code. Was "exact" before 2026-11-21.
Related:
SHAH-CAT-12Failure Handling & Implementation Robustness -
partial
-
MAST: Multi-Agent System Failure Taxonomy
Mapping drafted by an AI model from the source text; not yet reviewed by a person.
-
partial
FM-1.1Disobey task specification — FM-1.1 includes code that runs but misses a stated requirement (the paper's ChatDev Wordle example ships a fixed word list against the spec), which is logic_error. FM-1.1 also covers non-code violations.
Related:
FM-3.2No or incomplete verification -
partial
-
OWASP Top 10 for Agentic Applications 2026
Mapping drafted by an AI model from the source text; not yet reviewed by a person.
Related:
ASI02Tool Misuse and Exploitation;ASI08Cascading Failures -
Swiss Cheese Model for AI Safety: Multi-Layered Runtime Guardrail Taxonomy
Mapping drafted by an AI model from the source text; not yet reviewed by a person.
Related:
SC-QA-01Accuracy -
Taxonomy of Failure Mode in Agentic AI Systems (Microsoft AI Red Team, version 1.0)
Mapping drafted by an AI model from the source text; not yet reviewed by a person.
Related:
MS-EF-07Misinterpretation of instructions
Related failure modes
- Reinventing existing code (dependency blindness) — Hand-written code for what the standard library, a dependency or the codebase already provides.
- Cascading failures — Each fix introduces new errors and the code drifts further from working.
- Test manipulation — Tests edited, weakened or skipped so failures disappear instead of the bug being fixed.
- False “done” claims (incomplete execution) — The agent reports the task done while parts are skipped, stubbed or not working.
Research
Questions and answers
What is a logic error in AI-generated code?
A logic error is when an AI coding agent uses real, current APIs correctly but writes logic that computes the wrong result for the task, for example an inverted condition, an off-by-one or a faulty algorithm. It is the classic “passes the type check, fails correctness” bug. It differs from code hallucination, where the API does not exist, and from obsolescence, where the API is out of date. A fix that makes things worse is a cascading failure.
How do you detect logic errors in AI-generated code?
Signals that point to it: Unit tests fail with correctness assertions (not import or type errors). Execution in a sandbox produces outputs that contradict the task spec. Mutation testing reveals that substitution of logic primitives does not change test outcomes (test insensitivity is a different signal, but logic-error cases often co-occur). Symbolic or property-based tests find counterexamples. On SAICA-Bench: Best reviewer: mimo-v2.6-flash at J +74.3%. Its paired difference from the next row (deepseek-v4-flash) has a 95% interval that excludes 0.
Which reviewers catch logic errors in AI-generated code on SAICA-Bench, and how well?
Best reviewer: mimo-v2.6-flash at J +74.3%. Its paired difference from the next row (deepseek-v4-flash) has a 95% interval that excludes 0. The best catalog tool that ran its own check is PR-Agent, at J +53.2%. A rule with no understanding of the failure scores J +39.4% on the same set (Surface classifier), so read the scores against that, not against 0. Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.
How was detection of logic errors in AI-generated code measured?
On 2,440 scored samples: 1,536 wrong functions and 904 clean ones. Where the bad samples come from: model-written code, not agent output (11 code models of 2023-24 (CodeLlama, DeepSeek-Coder, StarCoder2, Llama 3, gpt-4o-mini) answering MBPP and HumanEval tasks; not agent runs). Label: execution: the function fails a test assertion of its task. Data: Collu-Bench (lt-asset, BSD-3-Clause-Clear): MBPP and HumanEval functions written by 11 code models. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.
Benchmark results from 2026-11-22. The definition, detection signals and tags come from the SAICA knowledge graph.