Spracklen et al. find ~20% of AI-generated code samples import nonexistent packages
USENIX Security 2024 study of 16 LLMs across 576K prompts: 19.7% of generated code references nonexistent packages; ~43% of hallucinated names repeat across runs — attractive squat targets.
Description
The Spracklen et al. paper "We Have a Package for You!" (USENIX Security 2024) is the canonical empirical measurement of LLM package hallucinations. Across 16 LLMs and 576,000 code samples, 19.7% of samples reference at least one nonexistent package. 43% of the hallucinated package names re-appear across multiple runs with similar prompts, making them predictable to adversaries. The paper formalizes "package hallucination" as the failure class underlying slopsquatting. It is the academic ground-truth for the `fabrication` × `supply_chain_attack` intersection that SAICA-KG models.
Exhibited failure modes
- Fabrication — Generating reference to non-existent APIs, packages, or symbols.
- Supply-chain attack — Slopsquatting, typosquatting, compromised MCP servers.
Affected systems
- GPT-4
- GPT-3.5
- CodeLlama
- DeepSeekCoder
- Mistral
- 11 others evaluated in the paper
Mitigated by
- Socket — Dependency supply-chain scanner that inspects packages before install.
- Snyk — Vulnerability scanning for code, dependencies, containers, and IaC.
Documented by
Sources
Notes
This entry models "measurement as incident" — a dataset-scale finding rather than a single event. `reproducibility: confirmed` because the methodology was peer-reviewed at USENIX Security.