DeepEval
14,938The open-source LLM evaluation framework for testing and evaluating LLM applications across 20+ custom and researched metrics.
Maturity tier details
Computed tier: 2B(raw score 4.67). Effective tier: 2B.
Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.67
Source of truth:pipeline/maturity/score.py
Description
DeepEval is an open-source framework for unit testing and evaluating LLM applications. It provides 20+ research-backed evaluation metrics including hallucination detection, correctness, answer relevancy, faithfulness, contextual recall/precision, and prompt injection detection. Developers can integrate it into CI/CD pipelines for continuous evaluation of LLM outputs. Supports custom metrics, chain-of-thought evaluation, and synthetic data generation for test cases.
Why it's included
Occupies detection × post_generation × full_hitl as a unit-testing harness for LLM applications: developers write pytest-style assertions over 20+ research-backed metrics (faithfulness, contextual recall/precision, hallucination, G-Eval chain-of-thought) and gate CI on them. Distinguished from observability-first peers (langfuse, langsmith) by centering test-case authoring rather than trace capture, and from braintrust by being open-source and Pytest-native. Useful as the canonical CI-gate exemplar for the post-generation evaluation cell.
Addresses failure modes
- Fabrication — Generating reference to non-existent APIs, packages, or symbols.
- Logic error — Correct API use, incorrect logic.