Skip to main content
SAICA-KG

Tools

lm-evaluation-harness

12,298

A framework for few-shot evaluation of language models.

DetectionPost-generationFull HITLStableTier 2BEnvironment
Maturity tier details

Computed tier: 2B(raw score 4.64). Effective tier: 2B.

Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.64

Source of truth:pipeline/maturity/score.py

Description

EleutherAI's canonical few-shot LLM evaluation framework. Implements 200+ standard benchmarks (MMLU, HellaSwag, ARC, TruthfulQA, HumanEval, GSM8K, BBH) in a unified harness. Backbone of Hugging Face's Open LLM Leaderboard.

Why it's included

Foundational LLM eval harness: cited as 'the harness' in leaderboard and model-release contexts. Differentiated from lighteval by a larger task catalog and older, broader adoption.

Addresses failure modes

  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Logic error — Correct API use, incorrect logic.

Techniques

few-shot-evalstandard-benchmarksreference-harness

View raw YAML on GitHub →