Skip to main content
SAICA-KG

Tools

helm

2,758

Holistic Evaluation of Language Models (HELM).

DetectionPost-generationFull HITLExperimentalTier 2BEnvironment
Maturity tier details

Computed tier: 2B(raw score 4.38). Effective tier: 2B.

Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.38

Source of truth:pipeline/maturity/score.py

Description

Stanford CRFM's HELM is a benchmark suite covering accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency across a scenario × metric matrix. Establishes the 'holistic evaluation' template that many later LLM evaluation efforts cite.

Why it's included

Canonical holistic-eval suite — represents the research branch of LLM evaluation that keeps safety / bias / robustness as first-class metrics alongside accuracy.

Addresses failure modes

  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Logic error — Correct API use, incorrect logic.
  • Scope creep — Actions outside declared task boundary.
  • Security vulnerability — Insecure patterns, injection flaws, weak cryptography, secret leakage.

Techniques

scenario-metric-matrixholistic-evalrobustness-scenarios

View raw YAML on GitHub →