Skip to main content
SAICA-KG

Recipes

Fabrication-resistant Python agent

context-hub (RAG grounding) → pydantic-ai (schema-validated outputs) → e2b (sandboxed execution) → pytest post-check. Fails fast on hallucinated symbols before any change leaves the sandbox.

case studiedpython

Description

A four-stage Python-agent composition whose ordering is designed to push every failure class the `fabrication` and `logic_error` FailureModes cover into a different observable signal. Stage 1 (pre-generation): context-hub retrieves relevant in-repo and standard-library context so the model generates against real symbols. Stage 2 (in-generation): pydantic-ai enforces Pydantic schemas on every tool call and return value, rejecting malformed outputs at parse time. Stage 3 (post-generation): e2b runs the generated code inside a fresh sandboxed container. `ImportError` / `AttributeError` on a fabricated symbol surfaces here, not in the user's repo. Stage 4 (post-execution): a pytest run over the generated module — the test suite is the final correctness gate and also catches logic errors that parsed and imported successfully. This is the canonical v0.1 reference composition for `python` + `graduated_hitl` workflows.

Stack

Ordered composition. Each stage consumes the output of the one before it.

  1. Stage 1
    Context Hub

    Up-to-date, curated API docs for coding agents.

  2. Stage 2
    PydanticAI

    Structured-output agent framework with Pydantic-typed tool and return schemas.

  3. Stage 3
    E2B

    Secure sandboxes for AI-generated code execution.

Targets failure modes

  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Logic error — Correct API use, incorrect logic.

Notes

Pytest is the post-check but not a separate node in the KG. Evidence tier `case_studied` reflects that the composition appears in Andrew Ng's context-hub reference materials and PydanticAI docs but has not been formally benchmarked.

View raw YAML on GitHub →