Skip to main content
SAICA-KG

Tools

evals

18,262

Framework for evaluating LLMs and LLM systems, plus an open registry of benchmarks.

DetectionPost-generationFull HITLStableTier 2AEnvironment
Maturity tier details

Computed tier: 2A(raw score 4.7). Effective tier: 2A.

Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.7

Source of truth:pipeline/maturity/score.py

Description

OpenAI Evals is a framework for evaluating LLMs and LLM systems and a community registry of benchmarks. Supports both traditional reference graders and model-graded evals. Distinct from lm-evaluation-harness by emphasizing user-contributed evals over a fixed benchmark suite.

Why it's included

Canonical 'bring-your-own-eval' registry: central artifact for OpenAI-model comparisons.

Addresses failure modes

  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Logic error — Correct API use, incorrect logic.
  • Scope creep — Actions outside declared task boundary.

Techniques

model-graded-evalseval-registryreference-graders

View raw YAML on GitHub →