Observability-first prototyping
cursor (IDE) → langfuse (trace / eval / dataset capture) → confident-ai-deepteam (scenario eval playground). Every prototype run emits structured traces the team reviews before promoting to...
Description
Composition targeting `context_pollution` and `logic_error` in early prototyping. The motivating pattern: teams ship vibe-coded features with no trace history, then cannot diagnose regressions. Stage 1: cursor remains the primary IDE / agent interface. Stage 2: langfuse captures every LLM call, tool invocation, prompt, and output as a structured trace. Traces double as dataset rows for downstream evaluation. Stage 3: confident-ai-deepteam replays captured traces against a scenario suite (adversarial prompts, edge-case inputs) as an evaluation gate before a change is promoted from prototype to production.
Stack
Ordered composition. Each stage consumes the output of the one before it.
- Stage 2DeepTeam
Open-source LLM red-teaming framework from Confident AI with 40+ attack vulnerabilities.
Targets failure modes
- Context pollution — Trajectory drift, hallucination spirals, overlong or off-task context.
- Logic error — Correct API use, incorrect logic.
Notes
"Evaluation playground" from the v0.1 spec maps to confident-ai-deepteam (scenario replay). `anecdotal` because while all three tools publish individual case studies, the composition is convention-based, not benchmarked end-to-end.