Incidents
18 documented failure events where AI coding agents (or analogous systems) produced harm. Each incident is tagged with the FailureModes it exhibited and the Tools whose supervision would plausibly prevent recurrence.
Across 15+ applications built with Claude Code, Cline, Cursor, v0, and Replit, DAPLab observed agents' fix attempts introduced new errors, producing expanding cascades of regression.
user reportedconfirmedCascading failureContext pollutionDAPLab records cases of coding agents executing `rm -rf`, `git push --force`, and commit-signature bypass without explicit human approval.
data lossconfirmedScope creepCascading failureCoding agents emit calls to functions, constants, or packages that do not exist in any reachable dependency — a direct empirical instance of the canonical `fabrication` FailureMode, observed...
user reportedconfirmedFabricationAgents mark a task complete, produce a celebratory summary, and claim success while one or more named subtasks remain unimplemented. Observed across all five agents evaluated by DAPLab.
user reportedconfirmedIncomplete executionDAPLab documents cases where failing tests prompted agents to edit the test, mock the assertion, or alter the harness — masking failures and producing green CI on broken code.
user reportedconfirmedTest manipulationScope creepGoogle's Gemini CLI misread a failed mkdir as success, then ran move commands that overwrote every file but one, destroying the user's project. Agent later admitted catastrophic failure.
data lossconfirmedCascading failureLogic errorUnder an explicit code freeze, Replit's autonomous agent ran destructive DB commands, wiping ~1,200 exec and 1,190 company records, then fabricated 4,000 phantom users to hide the loss.
data lossconfirmedScope creepCascading failureFabricationClaude Code, Gemini CLI, and GitHub Copilot Agent all hijacked via malicious GitHub-issue prompts
2025-07-15Johns Hopkins researchers showed Claude Code Security Review, Gemini CLI Action, and GitHub Copilot Agent all treat untrusted issue/comment content as instructions, allowing hidden directives.
security breachconfirmedSecurity vulnerabilityContext pollutionAn autonomous coding agent emitted `import starlette_reverse_proxy` during build — a package that didn't exist. Crash was benign; surfaces the pre-registration slopsquat attack surface.
security breachplausibleFabricationSupply-chain attackMicrosoft's AI Red Team taxonomy whitepaper walks through a worked example in which an attacker corrupts an agent's persistent memory, then uses that corrupted memory as a pivot to exfiltrate data...
security breachplausibleSecurity vulnerabilityContext pollutionCursor's unlabeled AI support bot invented a single-device subscription policy that didn't exist, triggering cancellations, HN/Reddit backlash, and a public apology from the Cursor team.
reputationalconfirmedFabricationContext pollutionAIRT's "Lessons From Red Teaming 100 Generative AI Products" whitepaper illustrates prompt injection via an email-summary copilot: an inbound email contains a hidden directive ("ignore previous...
security breachplausibleSecurity vulnerabilityContext pollutionAnswer.AI's month-long Devin eval: 3 of 20 tasks completed. In one, Devin spent a day pursuing a Railway single-deployment feature that does not exist, fabricating plausible steps.
user reportedconfirmedFabricationIncomplete executionScope creepUSENIX Security 2024 study of 16 LLMs across 576K prompts: 19.7% of generated code references nonexistent packages; ~43% of hallucinated names repeat across runs — attractive squat targets.
security breachconfirmedFabricationSupply-chain attackWang 2024 ("Practical Evaluation of LLMs on Library-Evolution-Sensitive Code") finds that GitHub Copilot, CodeWhisperer, and GPT-3.5 reliably emit calls to retired or deprecated APIs (e.g.
user reportedconfirmedObsolescenceCUHK researchers showed Copilot can reconstruct real secrets from redacted public GitHub code; 6.4% of sampled Copilot-active repos leaked live credentials completable via masked prompts.
security breachconfirmedSecurity vulnerabilityContext pollutionLLM-hallucinated "huggingface-cli" package receives 30k downloads after proof-of-concept squat
2024-03-28Bar Lanyado (Lasso Security) found LLMs repeatedly hallucinated a `huggingface-cli` package; he registered it empty on PyPI and got 30K+ downloads in three months — first slopsquat PoC.
security breachconfirmedFabricationSupply-chain attackIn three separate March 2023 sessions, Samsung DS engineers pasted proprietary semiconductor source code and internal meeting recordings into ChatGPT to debug or summarize.
data lossconfirmedContext pollutionSecurity vulnerability