Devin spends >24h hallucinating Railway features rather than recognizing platform limit
Answer.AI's month-long Devin eval: 3 of 20 tasks completed. In one, Devin spent a day pursuing a Railway single-deployment feature that does not exist, fabricating plausible steps.
Description
Answer.AI's public evaluation of Devin (Cognition AI's "AI software engineer") ran for a month across 20 representative tasks, of which Devin completed 3 satisfactorily, 3 inconclusively, and failed 14 outright. The team's most-cited individual failure is the Railway deployment: asked to deploy multiple apps to a single Railway deployment (which Railway does not support), Devin spent more than a day attempting approaches based on hallucinated Railway features rather than reporting the platform constraint. Subsidiary failures included generating spaghetti code for greenfield projects and producing a DaisyUI theme that "looked right" but used default rather than requested colors. The review is the most cited cautionary public study of autonomous coding-agent incompleteness.
Exhibited failure modes
- Fabrication — Generating reference to non-existent APIs, packages, or symbols.
- Incomplete execution — Agent claims completion while subtasks are skipped, stubbed, or silently dropped.
- Scope creep — Actions outside declared task boundary.
Affected systems
- Devin (Cognition AI)
Mitigated by
No mitigating tools indexed for this incident yet.
Sources
Notes
`mitigated_by` cites tools whose plan-mode / per-step review pattern would have surfaced the Railway limitation before a day of wasted autonomous iteration. Do not imply they would "fix" autonomy mismatch — they would degrade the task to graduated-HITL, not solve it.