Logic error
Correct API use, incorrect logic.
Description
The agent uses real, current APIs correctly at the signature level but encodes incorrect logic for the task. The canonical "passes type check, fails correctness" class. Distinct from fabrication (API doesn't exist) and obsolescence (wrong version).
Detection signals
- Unit tests fail with correctness assertions (not import or type errors).
- Execution in a sandbox produces outputs that contradict the task spec.
- Mutation testing reveals that substitution of logic primitives does not change test outcomes (test insensitivity is a different signal, but logic-error cases often co-occur).
- Symbolic or property-based tests find counterexamples.
Prior work
- Fang Liu (2024) — Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
- Zhu (2025) — Where LLM Agents Fail: A Taxonomy of Errors and the AgentErrorBench Dataset
Crosswalks
Agent accepts incorrect output without checking.
MAST Verification Failure covers acceptance of incorrect output; logic_error is a cause upstream.
Mitigating tools
- agenta — Open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and observability.
- Arize Phoenix — Open-source LLM tracing and evaluation platform built on OpenTelemetry.
- Braintrust — Commercial eval-first platform for LLM applications with prompt playground and online scoring.
- Daytona — Secure AI sandbox workspaces with sub-second start and stateful dev environments.
- DeepEval — The open-source LLM evaluation framework for testing and evaluating LLM applications across 20+ custom and researched metrics.
- docetl — LLM-powered data processing with automatic pipeline optimization.
- E2B — Secure sandboxes for AI-generated code execution.
- ESLint — Pluggable JavaScript/TypeScript linter and code-quality enforcer.
- giskard-oss — Open-source evaluation and testing library for LLM agents and ML models.
- Guardrails AI — Output validation framework with a catalog of validators and reask-on-failure loops.
- Guidance — Programmable interleaving of prompt, control flow, and constrained generation.
- Helicone — Open-source LLM observability proxy with one-line integration and caching.
- helm — Holistic Evaluation of Language Models (HELM).
- Instructor — Pydantic-typed structured output for LLM APIs with automatic retries.
- judgeval — The open source post-building layer for agents: evals, traces, and regressions.
- Langfuse — Open-source LLM engineering platform for tracing, evals, and prompt management.
- LangSmith — Hosted LLM observability, tracing, and evaluation platform.
- lighteval — Hugging Face all-in-one toolkit for evaluating LLMs across multiple backends.
- lm-evaluation-harness — A framework for few-shot evaluation of language models.
- lmms-eval — One-for-all multimodal evaluation toolkit across text, image, video, and audio tasks.
- lmnr — Laminar — open-source observability platform purpose-built for AI agents.
- logfire — Pydantic Logfire — observability for AI agents, LLMs, and Python apps built on OpenTelemetry.
- mlflow — Open-source platform for the machine learning lifecycle — experiments, tracking, models, and LLM evaluation.
- Modal — Serverless sandboxed runtime with gVisor-isolated containers for agent code execution.
- mypy — Static type checker for Python. Catches mismatched types before runtime.
- evals — Framework for evaluating LLMs and LLM systems, plus an open registry of benchmarks.
- opik — Debug, evaluate, and monitor LLM applications, RAG systems, and agentic workflows.
- Outlines — Constrained decoding via finite-state machines over regex, JSON Schema, and CFGs.
- pr-agent — AI-powered tool for automated pull-request analysis, feedback, and suggestions.
- promptbench — Unified evaluation framework for large language models with adversarial perturbations.
- promptflow — Build high-quality LLM apps — from prototyping, testing to production deployment and monitoring.
- promptfoo — Test prompts, agents, and RAGs — red-teaming, pentesting, and vulnerability scanning for LLMs.
- PydanticAI — Structured-output agent framework with Pydantic-typed tool and return schemas.
- Pyright — Microsoft's fast static type checker for Python.
- pytest — Python test framework — discipline gate for "done means tests pass."
- ragas — Supercharge your LLM application evaluations — RAG- and agent-centric metrics.
- Ruff — Extremely fast Python linter and formatter, written in Rust.
- Smolagents — Minimalist code-agent library that expresses actions as executed Python.
- uqlm — Uncertainty Quantification for Language Models.