Skip to main content
SAICA-KG

Incidents

Code-generation LLMs emit obsolete Java 8 collection APIs despite modern project target

Wang 2024 ("Practical Evaluation of LLMs on Library-Evolution-Sensitive Code") finds that GitHub Copilot, CodeWhisperer, and GPT-3.5 reliably emit calls to retired or deprecated APIs (e.g.

2024-06-01user reportedconfirmed

Description

Wang, Zhang et al. 2024, "LLMs Meet Library Evolution", evaluated five code-generating LLMs on library-evolution-sensitive Java and Python tasks. In ~30% of evaluated completions the generated code used APIs that had been deprecated or removed in the library version specified by the project's build metadata. Representative Java examples included `Vector`, `Hashtable`, and older `Collections.synchronized*` patterns that modern `java.util.concurrent` APIs have superseded. The paper established that obsolescence is not random: LLMs overweight older API surface because training corpora skew toward older, more-copied code. SAICA-KG records the study as the empirical anchor for the `obsolescence` FailureMode.

Exhibited failure modes

  • Obsolescence — Using deprecated APIs, retired packages, or outdated library versions — often because the model's training data pre-dates current releases.

Affected systems

  • GitHub Copilot
  • Amazon CodeWhisperer
  • GPT-3.5
  • CodeLlama

Mitigated by

  • Semgrep — Fast, rule-based static analysis with a pattern syntax that mirrors source code.

Documented by

Sources

Notes

Date approximated to arxiv v1 month. Mitigations cite sourcegraph-cody (graph-aware retrieval of the current API surface) and semgrep (deprecation rules at commit time).

View raw YAML on GitHub →