Skip to main content
SAICA-KG

Tools

lighteval

2,387

Hugging Face all-in-one toolkit for evaluating LLMs across multiple backends.

DetectionPost-generationFull HITLExperimentalTier 2BEnvironment
Maturity tier details

Computed tier: 2B(raw score 4.35). Effective tier: 2B.

Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.35

Source of truth:pipeline/maturity/score.py

Description

Hugging Face's lightweight LLM evaluation toolkit. Supports multiple inference backends (vLLM, Accelerate, nanotron) and is the backbone of the Hugging Face Open LLM Leaderboard v2. Positioned as a leaner, HF-native successor to lm-evaluation-harness.

Why it's included

HF-native leaderboard harness. Differentiated from lm-evaluation-harness by tighter vLLM/Accelerate integration and faster eval runs.

Addresses failure modes

  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Logic error — Correct API use, incorrect logic.

Techniques

multi-backend-evalopen-llm-leaderboard-v2

View raw YAML on GitHub →