Skip to main content
SAICA-KG

Tools

promptbench

2,801

Unified evaluation framework for large language models with adversarial perturbations.

DetectionPost-generationFull HITLExperimentalTier 3PromptEnvironment
Maturity tier details

Computed tier: 3(raw score 3.88). Effective tier: 3.

Facets:documented_in=0evaluated_on=0integration_surfaces=0.5paper_citations=0recent_commit=2stars=1.38

Source of truth:pipeline/maturity/score.py

Description

Microsoft PromptBench is a framework to benchmark LLM robustness under adversarial prompt perturbations. Evaluates stability, robustness, and reliability with defined attack taxonomies and measures prompt sensitivity of outputs.

Why it's included

Robustness-under-perturbation cell of LLM evaluation — pairs well with garak/promptmap (security) and helm (holistic).

Addresses failure modes

  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Security vulnerability — Insecure patterns, injection flaws, weak cryptography, secret leakage.
  • Logic error — Correct API use, incorrect logic.

Techniques

adversarial-perturbationsprompt-sensitivity-metrics

View raw YAML on GitHub →