opik
18,988Debug, evaluate, and monitor LLM applications, RAG systems, and agentic workflows.
Maturity tier details
Computed tier: 2A(raw score 4.71). Effective tier: 2A.
Facets:documented_in=0evaluated_on=0integration_surfaces=1paper_citations=0recent_commit=2stars=1.71
Source of truth:pipeline/maturity/score.py
Description
Opik is Comet ML's open-source platform for debugging, evaluating, and monitoring LLM applications, RAG pipelines, and multi-agent workflows. Provides trace capture, LLM-as-judge graders, test dataset management, and production monitoring dashboards. Self-hostable alongside a SaaS offering.
Why it's included
Occupies detection × post_generation × full_hitl. Differentiated from langfuse by being Comet-ecosystem-integrated (shared ML experiment-tracking backbone) and from phoenix by its agent-focused trace model.
Addresses failure modes
- Fabrication — Generating reference to non-existent APIs, packages, or symbols.
- Logic error — Correct API use, incorrect logic.
- Scope creep — Actions outside declared task boundary.
- Incomplete execution — Agent claims completion while subtasks are skipped, stubbed, or silently dropped.