Skip to main content
SAICA-KG

Tools

lmms-eval

4,052

One-for-all multimodal evaluation toolkit across text, image, video, and audio tasks.

DetectionPost-generationFull HITLExperimentalTier 2BEnvironment
Maturity tier details

Computed tier: 2B(raw score 4.44). Effective tier: 2B.

Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.44

Source of truth:pipeline/maturity/score.py

Description

LMMs-Eval is the EvolvingLMMs-Lab's multimodal evaluation harness — a multimodal counterpart to lm-evaluation-harness. Implements 100+ benchmarks across text, image, video, and audio tasks in a consistent interface.

Why it's included

Fills the multimodal-eval cell. Differentiated from lm-evaluation-harness (text-only) and lighteval by its native multimodal scope.

Addresses failure modes

  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Logic error — Correct API use, incorrect logic.

Techniques

multimodal-benchmarksunified-harness

View raw YAML on GitHub →