Skip to main content
SAICA-KG

Papers

Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities

Arjun Krishna, Erick Galinkin, Leon Derczynski, Jeffrey Martin · 2025 · arXiv

Source →arXiv:2501.19012DOI: 10.48550/arXiv.2501.190127 citations
benchmarksupply-chain

TL;DR

This work analyzes package hallucination behaviour in LLMs across popular programming languages examining both existing package references and fictional dependencies, finding an inverse correlation between package hallucination rate and the HumanEval coding benchmark, offering a heuristic for evaluating the propensity of a model to hallucinate packages.

Abstract

Large Language Models (LLMs) have become an essential tool in the programmer's toolkit, but their tendency to hallucinate code can be used by malicious actors to introduce vulnerabilities to broad swathes of the software supply chain. In this work, we analyze package hallucination behaviour in LLMs across popular programming languages examining both existing package references and fictional dependencies. By analyzing this package hallucination behaviour we find potential attacks and suggest defensive strategies to defend against these attacks. We discover that package hallucination rate is predicated not only on model choice, but also programming language, model size, and specificity of the coding task request. The Pareto optimality boundary between code generation performance and package hallucination is sparsely populated, suggesting that coding models are not being optimized for secure code. Additionally, we find an inverse correlation between package hallucination rate and the HumanEval coding benchmark, offering a heuristic for evaluating the propensity of a model to hallucinate packages. Our metrics, findings and analyses provide a base for future models, securing AI-assisted software development workflows against package supply chain attacks.

Editorial notes

Measures package-hallucination behavior across multiple LLMs and programming languages. Two key empirical claims for SAICA-KG: (a) the Pareto frontier between code-generation performance and package-hallucination rate is sparsely populated, suggesting coding models are not currently optimized for secure code, and (b) there is an inverse correlation between package-hallucination rate and HumanEval performance, offering a heuristic for forecasting which models are most exposed. Relates to SAICA-KG failure modes: supply_chain_attack, fabrication, security_vulnerability. Cite as a complement to spracklen-2024 — same problem, different measurement angle.

View raw YAML on GitHub →