judgeval
1,024The open source post-building layer for agents: evals, traces, and regressions.
DetectionPost-generationFull HITLExperimentalTier 2BEnvironment
Maturity tier details
Computed tier: 2B(raw score 4.2). Effective tier: 2B.
Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.2
Source of truth:pipeline/maturity/score.py
Description
Judgeval is an open-source 'post-building' evaluation layer for agents from Judgment Labs. Pairs trace capture with regression testing, environment-as-data, and LLM-as-judge graders. Designed to be added after an agent is built to monitor quality and catch regressions in CI.
Why it's included
Fills the 'CI regression harness for agents' niche. Distinct from promptfoo (prompt-level regression) by operating on full agent runs and environment data.
Addresses failure modes
- Fabrication — Generating reference to non-existent APIs, packages, or symbols.
- Scope creep — Actions outside declared task boundary.
- Logic error — Correct API use, incorrect logic.
- Test manipulation — Agent gaming its own evaluation — editing tests or test harness so failures disappear without fixing the underlying bug.
Techniques
agent-regression-testsllm-as-judgeenvironment-as-data