Skip to main content
SAICA-KG

Recipes

Observability-first prototyping

cursor (IDE) → langfuse (trace / eval / dataset capture) → confident-ai-deepteam (scenario eval playground). Every prototype run emits structured traces the team reviews before promoting to...

anecdotalpolyglot

Description

Composition targeting `context_pollution` and `logic_error` in early prototyping. The motivating pattern: teams ship vibe-coded features with no trace history, then cannot diagnose regressions. Stage 1: cursor remains the primary IDE / agent interface. Stage 2: langfuse captures every LLM call, tool invocation, prompt, and output as a structured trace. Traces double as dataset rows for downstream evaluation. Stage 3: confident-ai-deepteam replays captured traces against a scenario suite (adversarial prompts, edge-case inputs) as an evaluation gate before a change is promoted from prototype to production.

Stack

Ordered composition. Each stage consumes the output of the one before it.

  1. Stage 1
    Langfuse

    Open-source LLM engineering platform for tracing, evals, and prompt management.

  2. Stage 2
    DeepTeam

    Open-source LLM red-teaming framework from Confident AI with 40+ attack vulnerabilities.

Targets failure modes

  • Context pollution — Trajectory drift, hallucination spirals, overlong or off-task context.
  • Logic error — Correct API use, incorrect logic.

Notes

"Evaluation playground" from the v0.1 spec maps to confident-ai-deepteam (scenario replay). `anecdotal` because while all three tools publish individual case studies, the composition is convention-based, not benchmarked end-to-end.

View raw YAML on GitHub →