Skip to main content
SAICA-KG

Tools

DeepEval

14,938

The open-source LLM evaluation framework for testing and evaluating LLM applications across 20+ custom and researched metrics.

DetectionPost-generationFull HITLExperimentalTier 2BEnvironment
Maturity tier details

Computed tier: 2B(raw score 4.67). Effective tier: 2B.

Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.67

Source of truth:pipeline/maturity/score.py

Description

DeepEval is an open-source framework for unit testing and evaluating LLM applications. It provides 20+ research-backed evaluation metrics including hallucination detection, correctness, answer relevancy, faithfulness, contextual recall/precision, and prompt injection detection. Developers can integrate it into CI/CD pipelines for continuous evaluation of LLM outputs. Supports custom metrics, chain-of-thought evaluation, and synthetic data generation for test cases.

Why it's included

Occupies detection × post_generation × full_hitl as a unit-testing harness for LLM applications: developers write pytest-style assertions over 20+ research-backed metrics (faithfulness, contextual recall/precision, hallucination, G-Eval chain-of-thought) and gate CI on them. Distinguished from observability-first peers (langfuse, langsmith) by centering test-case authoring rather than trace capture, and from braintrust by being open-source and Pytest-native. Useful as the canonical CI-gate exemplar for the post-generation evaluation cell.

Addresses failure modes

  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Logic error — Correct API use, incorrect logic.

View raw YAML on GitHub →