Skip to main content
SAICA-KG

Tools

judgeval

1,024

The open source post-building layer for agents: evals, traces, and regressions.

DetectionPost-generationFull HITLExperimentalTier 2BEnvironment
Maturity tier details

Computed tier: 2B(raw score 4.2). Effective tier: 2B.

Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.2

Source of truth:pipeline/maturity/score.py

Description

Judgeval is an open-source 'post-building' evaluation layer for agents from Judgment Labs. Pairs trace capture with regression testing, environment-as-data, and LLM-as-judge graders. Designed to be added after an agent is built to monitor quality and catch regressions in CI.

Why it's included

Fills the 'CI regression harness for agents' niche. Distinct from promptfoo (prompt-level regression) by operating on full agent runs and environment data.

Addresses failure modes

  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Scope creep — Actions outside declared task boundary.
  • Logic error — Correct API use, incorrect logic.
  • Test manipulation — Agent gaming its own evaluation — editing tests or test harness so failures disappear without fixing the underlying bug.

Techniques

agent-regression-testsllm-as-judgeenvironment-as-data

View raw YAML on GitHub →