Skip to main content
SAICA-KG

Incidents

Spracklen et al. find ~20% of AI-generated code samples import nonexistent packages

USENIX Security 2024 study of 16 LLMs across 576K prompts: 19.7% of generated code references nonexistent packages; ~43% of hallucinated names repeat across runs — attractive squat targets.

2024-08-14security breachconfirmed

Description

The Spracklen et al. paper "We Have a Package for You!" (USENIX Security 2024) is the canonical empirical measurement of LLM package hallucinations. Across 16 LLMs and 576,000 code samples, 19.7% of samples reference at least one nonexistent package. 43% of the hallucinated package names re-appear across multiple runs with similar prompts, making them predictable to adversaries. The paper formalizes "package hallucination" as the failure class underlying slopsquatting. It is the academic ground-truth for the `fabrication` × `supply_chain_attack` intersection that SAICA-KG models.

Exhibited failure modes

  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Supply-chain attack — Slopsquatting, typosquatting, compromised MCP servers.

Affected systems

  • GPT-4
  • GPT-3.5
  • CodeLlama
  • DeepSeekCoder
  • Mistral
  • 11 others evaluated in the paper

Mitigated by

  • Socket — Dependency supply-chain scanner that inspects packages before install.
  • Snyk — Vulnerability scanning for code, dependencies, containers, and IaC.

Documented by

Sources

Notes

This entry models "measurement as incident" — a dataset-scale finding rather than a single event. `reproducibility: confirmed` because the methodology was peer-reviewed at USENIX Security.

View raw YAML on GitHub →