evals
18,262Framework for evaluating LLMs and LLM systems, plus an open registry of benchmarks.
DetectionPost-generationFull HITLStableTier 2AEnvironment
Maturity tier details
Computed tier: 2A(raw score 4.7). Effective tier: 2A.
Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.7
Source of truth:pipeline/maturity/score.py
Description
OpenAI Evals is a framework for evaluating LLMs and LLM systems and a community registry of benchmarks. Supports both traditional reference graders and model-graded evals. Distinct from lm-evaluation-harness by emphasizing user-contributed evals over a fixed benchmark suite.
Why it's included
Canonical 'bring-your-own-eval' registry: central artifact for OpenAI-model comparisons.
Addresses failure modes
- Fabrication — Generating reference to non-existent APIs, packages, or symbols.
- Logic error — Correct API use, incorrect logic.
- Scope creep — Actions outside declared task boundary.
Techniques
model-graded-evalseval-registryreference-graders