helm
2,758Holistic Evaluation of Language Models (HELM).
DetectionPost-generationFull HITLExperimentalTier 2BEnvironment
Maturity tier details
Computed tier: 2B(raw score 4.38). Effective tier: 2B.
Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.38
Source of truth:pipeline/maturity/score.py
Description
Stanford CRFM's HELM is a benchmark suite covering accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency across a scenario × metric matrix. Establishes the 'holistic evaluation' template that many later LLM evaluation efforts cite.
Why it's included
Canonical holistic-eval suite — represents the research branch of LLM evaluation that keeps safety / bias / robustness as first-class metrics alongside accuracy.
Addresses failure modes
- Fabrication — Generating reference to non-existent APIs, packages, or symbols.
- Logic error — Correct API use, incorrect logic.
- Scope creep — Actions outside declared task boundary.
- Security vulnerability — Insecure patterns, injection flaws, weak cryptography, secret leakage.
Techniques
scenario-metric-matrixholistic-evalrobustness-scenarios