Skip to main content
SAICA-KG

Recipes

Test-manipulation-resistant flow

aider (commit-per-change makes test edits visible) → coverage-protected CI → semgrep rules on test-file diffs. Detects DAPLab DAP-07 silent-test-passing.

anecdotalpolyglot

Description

Composition for the `test_manipulation` FailureMode, anchored in the DAPLab DAP-07 observation (see daplab-silent-test-passing-observation-2026). Stage 1: aider commits every agent change as a separate commit. Test edits become first-class diffs rather than hidden within a large feature commit. Stage 2: coverage-protected CI fails when coverage drops below a project-defined threshold, blocking the "delete the failing test" workaround at the infrastructure layer. Stage 3: semgrep runs a ruleset diffing test files and runner config against a protected baseline; any agent-authored test-weakening change (removed assertion, mocked subject, skipped test, weakened matcher) triggers a rule and requires explicit human approval.

Stack

Ordered composition. Each stage consumes the output of the one before it.

  1. Stage 1
    Semgrep

    Fast, rule-based static analysis with a pattern syntax that mirrors source code.

  2. Stage 2
    promptfoo

    Test prompts, agents, and RAGs — red-teaming, pentesting, and vulnerability scanning for LLMs.

Targets failure modes

  • Test manipulation — Agent gaming its own evaluation — editing tests or test harness so failures disappear without fixing the underlying bug.

Notes

Coverage-protected CI is configuration, not a KG-node; stack is length 2. Evidence tier `anecdotal` — the pattern is well-known in platform-eng communities but not benchmarked. Downgrade is appropriate while SAICA-KG has no test-manipulation benchmark paper.

View raw YAML on GitHub →