Code-generation LLMs emit obsolete Java 8 collection APIs despite modern project target
Wang 2024 ("Practical Evaluation of LLMs on Library-Evolution-Sensitive Code") finds that GitHub Copilot, CodeWhisperer, and GPT-3.5 reliably emit calls to retired or deprecated APIs (e.g.
Description
Wang, Zhang et al. 2024, "LLMs Meet Library Evolution", evaluated five code-generating LLMs on library-evolution-sensitive Java and Python tasks. In ~30% of evaluated completions the generated code used APIs that had been deprecated or removed in the library version specified by the project's build metadata. Representative Java examples included `Vector`, `Hashtable`, and older `Collections.synchronized*` patterns that modern `java.util.concurrent` APIs have superseded. The paper established that obsolescence is not random: LLMs overweight older API surface because training corpora skew toward older, more-copied code. SAICA-KG records the study as the empirical anchor for the `obsolescence` FailureMode.
Exhibited failure modes
- Obsolescence — Using deprecated APIs, retired packages, or outdated library versions — often because the model's training data pre-dates current releases.
Affected systems
- GitHub Copilot
- Amazon CodeWhisperer
- GPT-3.5
- CodeLlama
Mitigated by
- Semgrep — Fast, rule-based static analysis with a pattern syntax that mirrors source code.
Documented by
Sources
Notes
Date approximated to arxiv v1 month. Mitigations cite sourcegraph-cody (graph-aware retrieval of the current API surface) and semgrep (deprecation rules at commit time).