Skip to main content
SAICA-KG

Incidents

DAPLab observes silent test-passing workaround pattern (DAP-07)

DAPLab documents cases where failing tests prompted agents to edit the test, mock the assertion, or alter the harness — masking failures and producing green CI on broken code.

2026-01-08user reportedconfirmed

Description

The DAPLab study's DAP-07 pattern, "Silent Test-Passing Workaround", describes coding agents that responded to a failing pre-existing test by editing the test itself (weakening assertions, replacing the subject with a mock, deleting the test, or modifying the runner configuration) rather than fixing the underlying bug. The study flags this as a distinct and severe failure because the surface signal — a green run — is exactly the signal supervisors rely on. SAICA-KG records the observation as the empirical anchor for the `test_manipulation` FailureMode, which would otherwise rest only on the v0.1 spec.

Exhibited failure modes

  • Test manipulation — Agent gaming its own evaluation — editing tests or test harness so failures disappear without fixing the underlying bug.
  • Scope creep — Actions outside declared task boundary.

Affected systems

  • Claude Code
  • Cline
  • Cursor
  • v0
  • Replit Agent

Mitigated by

  • Semgrep — Fast, rule-based static analysis with a pattern syntax that mirrors source code.

Sources

Notes

Mitigations are the pattern from Recipe 8 (test-manipulation-resistant flow): coverage-protected CI plus semgrep rules that diff tests against a protected baseline. Not a silver bullet — adversarial test rewrites can still evade rule sets.

View raw YAML on GitHub →