lighteval
2,387Hugging Face all-in-one toolkit for evaluating LLMs across multiple backends.
DetectionPost-generationFull HITLExperimentalTier 2BEnvironment
Maturity tier details
Computed tier: 2B(raw score 4.35). Effective tier: 2B.
Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.35
Source of truth:pipeline/maturity/score.py
Description
Hugging Face's lightweight LLM evaluation toolkit. Supports multiple inference backends (vLLM, Accelerate, nanotron) and is the backbone of the Hugging Face Open LLM Leaderboard v2. Positioned as a leaner, HF-native successor to lm-evaluation-harness.
Why it's included
HF-native leaderboard harness. Differentiated from lm-evaluation-harness by tighter vLLM/Accelerate integration and faster eval runs.
Addresses failure modes
- Fabrication — Generating reference to non-existent APIs, packages, or symbols.
- Logic error — Correct API use, incorrect logic.
Techniques
multi-backend-evalopen-llm-leaderboard-v2