Papers
Academic and gray-literature references grounding SAICA-KG nodes. Each Tool, FailureMode, and Taxonomy that relies on a paper carries an explicit link.
AVDA: Autonomous Vibe Detection Authoring for Cybersecurity
Fatih Bulut, C. Depaolis, Raghav Batta, Anjali Mangal · 2026 · arXiv
AVDA is introduced, a framework that leverages the Model Context Protocol to automate detection authoring by integrating organizational context - existing detections, telemetry schemas, and style guides - into AI-assisted code generation by integrating seamlessly into standard developer environments.
agentic-aiMCPCharacterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes
Mehil B. Shah, Mohammad Mehdi Morovati, Mohammad Masudur Rahman, Foutse Khomh · 2026 · arXiv
385-fault grounded-theory study over 40 agentic-AI repos yields 37 fault types in 13 categories, 13 symptom classes, and 12 root-cause categories, validated by a 145-practitioner developer study.
failure-mode-taxonomyagentic-aiempiricalexternal-taxonomy-seedgrounded-theorySecurity Considerations for Artificial Intelligence Agents
Ninghui Li, Kaiyuan Zhang, Kyle Polley, Jerry Ma · 2026 · arXiv
This article details observations and recommendations concerning the security of frontier AI agents, and identifies standards and research gaps, including adaptive security benchmarks, policy models for delegation and privilege control, and guidance for secure multi-agent system design aligned with NIST risk management principles.
failure-mode-taxonomyagentic-aibenchmarkguardrailsAgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering
Rajesh Kumar, Waqar Ali, Junaid Ahmed, Najmah Ali, Shaban Usman · 2026 · arXiv
Introduces execution-grounded verification as a first-class principle: every code change must survive sandboxed execution before propagation. AGENTFORGE instantiates this with Planner/Coder/Tester/Debugger/Critic agents sharing memory and a mandatory Docker sandbox, achieving 40.0% on SWE-Bench Lite — 26-28 points above single-agent baselines.
multi-agentexecution-groundedsandboxingcoding-agentcorrectionverificationWhere Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
Ramtin Ehsani, Sakshi Pathak, Sarthak Rawal, Abdullah Mujahid, M. M. Imran, Preetha Chatterjee · 2026 · arXiv
Large-scale study of 33k agent-authored PRs across five coding agents on GitHub. Quantifies merge outcomes by task type, code-change size, CI, and review dynamics; derives a hierarchical taxonomy of rejection patterns from qualitative analysis of 600 PRs (duplicate PRs, unwanted features, lack of reviewer engagement, agent misalignment).
failure-mode-taxonomycoding-agentgithubempiricalagent-misalignmentPRBench: End-to-end Paper Reproduction in Physics Research
Shi Qiu, Jun Deng, Yiwei Deng, Haoran Dong, Jie Fu, Mao Li, Zeyu Li, Zhao Zhang, Huiwen Zheng, Lei Bao, Anqiang Lv, Zihan Mo, Yadi Niu, Yiyang Peng, Yu Tian, Yilin Wang, Ziyu Wang, Ziyu Wang, Jiashen Wei, Liu Wu, Ao Xue, Leyi Yang, Guanglu Yuan, Xia Zhan, Jingjun Zhang, Zifan Zheng, Pengfei Liu, Li Zhen, Kaiyang Li, Qichang Li, Ziheng Zhou, Guokui Nian, Yu-Xiao Xiao, Qing-Hong Cao, L. Dai, Xu Feng, Pengyu Gao, Ying Gu, Chang Liu, Jia Liu, M. Luo, Yan-Qing Ma, Liangchen Peng, Huichao Song, Shufeng Wang, Chenxu Wang, Tao Wang, Yi-Nan Wang, C. Wu, Peng Zhao, H. Zhu · 2026 · arXiv
This work evaluates a set of coding agents on PRBench and analyzes their capabilities across key dimensions of scientific reasoning and execution, identifying systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data.
failure-mode-taxonomycoding-agentbenchmarkguardrailsPackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
Xiting Liu, Yuetong Liu, Yitong Zhang, Jia Li, Shiyuan Hu · 2026 · arXiv
Extensive experiments on five widely used LLMs demonstrate that PackMonitor is a training-free, plug-and-play solution that consistently reduces package hallucination rates to zero while maintaining low-latency inference and preserving original model capabilities.
supervisionsupply-chainSkillReducer: Optimizing LLM Agent Skills for Token Efficiency
Yudong Gao, Zongjie Li, Yuanyuanyuan, Zimo Ji, Pingchuan Ma, Shuai Wang · 2026 · arXiv
A two-stage optimization framework that achieves a less-is-more effect where removing non-essential content reduces distraction in the context window, and generalize to an independent agent framework is presented.
failure-mode-taxonomycoding-agentagentic-aibenchmarkempiricalRIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics
Zhengyang Qi, Charles Dickens, Derek Pham, A. Dsouza, Armin Parchami, Frederic Sala, P. Varma · 2026 · arXiv
Grounded-theory taxonomy of 8 rubric-evaluation failure modes across 3 categories (reliability, content validity, consequential validity), developed from rubrics in instruction following, code generation, creative writing, and deep research; paired with automated diagnostic metrics.
failure-mode-taxonomyevaluationrubricgrounded-theorymulti-domainImporting Phantoms: Measuring LLM Package Hallucination Vulnerabilities
Arjun Krishna, Erick Galinkin, Leon Derczynski, Jeffrey Martin · 2025 · arXiv
This work analyzes package hallucination behaviour in LLMs across popular programming languages examining both existing package references and fictional dependencies, finding an inverse correlation between package hallucination rate and the HumanEval coding benchmark, offering a heuristic for evaluating the propensity of a model to hallucinate packages.
benchmarksupply-chainFault-Tolerant Sandboxing for AI Coding Agents: A Transactional Approach to Safe Autonomous Execution
B. Yan · 2025 · arXiv
Fault-tolerant sandboxing framework wrapping agent actions in atomic transactions over a policy-based interception layer plus a transactional filesystem snapshot mechanism. Demonstrates 100% interception of high-risk commands and 100% rollback of failed states at 14.5% overhead, contrasted against Gemini-CLI's headless-incompatible auth model.
sandboxingrecoverycoding-agentdestructive-actioncorrectionenvironmentPractical challenges of control monitoring in frontier AI deployments
David Lindner, Charlie Griffin, Tomasz Korbak, Roland S. Zimmermann, Geoffrey Irving, Sebastian Farquhar, Alan Cooney · 2025 · arXiv
Analyzes three forms of automated control monitoring with different latency-safety trade-offs: synchronous, semi-synchronous, and asynchronous. Introduces a high-level safety-case sketch and identifies three core challenges (oversight, latency, recovery) across four future-deployment case studies.
monitoringtemporal-phasecontrol-paradigmsupervision-vocabularysafety-caseDistinguishing Control and Oversight in Agentic AI
David Manheim, James Homewood · 2025 · arXiv
Distinguishes control (ex-ante / operational) from oversight (ex-post / governance). Provides vocabulary that converges with SAICA-KG's TemporalPhase and ControlParadigm partitions.
supervision-vocabularycontrol-paradigmtemporal-phaseMCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents
Dongsen Zhang, Zekun Li, Xu Luo, Xuannan Liu, Peipei Li, Wenjun Xu · 2025 · arXiv
MSB (MCP Security Benchmark), the first end-to-end evaluation suite that systematically measures how well LLM agents resist MCP-specific attacks throughout the full tool-use pipeline: task planning, tool invocation, and response handling, is presented.
failure-mode-taxonomyagentic-aiMCPbenchmarkNetMCP: Network-Aware Model Context Protocol Platform for LLM Capability Extension
Enhan Li, Hongyang Du, Kaibin Huang · 2025 · arXiv
Results show that SONAR consistently improves task success rate and reduces completion time and failure number compared with semantic-only, LLM-based baselines, demonstrating the value of network-aware design for production-scale LLM systems.
MCPbenchmarkCKGFuzzer: LLM-Based Fuzz Driver Generation Enhanced By Code Knowledge Graph
Hanxiang Xu, Wei Ma, Ti Zhou, Yanjie Zhao, Kai Chen, Qiang Hu, Yang Liu, Haoyu Wang · 2025 · ICSE Companion 2025 (IEEE/ACM 47th ICSE-Companion)
Automated fuzz-driver generation driven by an interprocedural code knowledge graph of the target repository and an LLM agent. Achieves 8.73% mean coverage improvement over SOTA, reduces crash-triage workload by 84.4%, and finds 11 real bugs (9 previously unreported).
code-knowledge-graphfuzzingsecurityagent-toolexecution-based-verificationPAGENT: Learning to Patch Software Engineering Agents
Haoran Xue, Gias Uddin, Song Wang · 2025 · arXiv
Empirical study of 769 failed patches from 7 SWE-bench coding agents over 114 unresolved issues; derives a 6-category failure taxonomy and introduces PAGENT, a static-analysis + LLM patch-repair tool that fixes 29/127 type-related failures.
failure-mode-taxonomycoding-agentswe-benchempiricalpatch-repairDecoding the Configuration of AI Coding Agents: Insights from Claude Code Projects
Helio Victor F. Santos, V. Costa, João Eduardo Montandon, Marco Túlio Valente · 2025 · arXiv
An empirical study of the configuration ecosystem of Claude Code, one of the most widely used agentic coding systems, to identify the software engineering concerns and practices they specify and how these concerns co-occur within individual files.
coding-agentagentic-aiempiricalAI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities
Huanting Wang, Jingzhi Gong, Huawei Zhang, Zheng Wang · 2025 · arXiv
AI agentic programming is an emerging paradigm where large language model (LLM)-based coding agents autonomously plan, execute, and interact with tools such as compilers, debuggers, and version control systems. Unlike conventional code generation, these agents decompose goals, coordinate multi-step processes, and adapt based on feedback, reshaping software development practices.
failure-mode-taxonomycoding-agentagentic-aisupervisionbenchmarksurveyAgentic Refactoring: An Empirical Study of AI Coding Agents
Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, Ahmed E. Hassan · 2025 · arXiv
A large-scale study of AI agent-generated refactorings in real-world open-source Java projects, analyzing 15,451 refactoring instances across 12,256 pull requests and 14,988 commits derived from the AIDev dataset shows that refactoring is a common and intentional activity in this development paradigm.
coding-agentagentic-aibenchmarkempiricalResearch Directions in Software Supply Chain Security
Laurie Williams, Giacomo Benedetti, Sivana Hamer, Ranindya Paramitha, Imranur Rahman, Mahzabin Tamanna, Greg Tystahl, Nusrat Zahan, Patrick Morrison, Yasemin Acar, Michel Cukier, Christian Kastner, Alexandros Kapravelos, Dominik Wermke, William Enck · 2025 · ACM Transactions on Software Engineering and Methodology
TOSEM roadmap paper on software supply-chain security: organizes attack vectors across dependencies, build infrastructure, and human factors; proposes a future research agenda. 54 citations.
supply-chain-securitysoftware-engineeringroadmapfailure-mode-seedLibrary Hallucinations in LLMs: Risk Analysis Grounded in Developer Queries
Lukas Twist, Jie M. Zhang, Mark Harman, Helen Yannakoudakis · 2025 · arXiv
The first systematic study of how user-level prompt variations impact library hallucinations in LLM-generated code reveals systemic vulnerabilities and underscores the fragility of LLMs to natural prompt variation.
supply-chainARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative Refinement
Manish Bhattarai, Miguel Cordova, Javier E. Santos, Dan O'Malley · 2025 · arXiv
ARS provides a practical, reproducible approach to reliable code synthesis using existing LLM checkpoints using existing LLM checkpoints as a state-action process with provable guarantees on termination, monotonic improvement, and bounded cost.
agentic-aiSecure or Suspect? Investigating Package Hallucinations of Shell Command in Original and Quantized LLMs
Md Nazmul Haque, Elizabeth Lin, L. Arkoh, B. Tadesse, Bowen Xu · 2025 · arXiv
This work conducts the first systematic empirical study of the impact of quantization on package hallucination and vulnerability risks in LLM-generated Go packages and reveals that most fabricated packages resemble realistic URL-based Go module paths, highlighting a systematic pattern in how LLMs hallucinate dependencies.
supply-chainempiricalWhy Do Multi-Agent LLM Systems Fail? MAST: A Multi-Agent System Failure Taxonomy
Melis Cemri, et al. · 2025 · arXiv
Grounded-theory taxonomy of 14 multi-agent failure modes across 3 clusters, derived from 1,600 annotated traces over 7 MAS frameworks.
failure-mode-taxonomymulti-agentexternal-taxonomy-seedCode Copycat Conundrum: Demystifying Repetition in LLM-based Code Generation
Mingwei Liu, Juntao Li, Ying Wang, Xueying Du, Zuoyu Ou, Qiuyuan Chen, Bingxu An, Zhao Wei, Yong Xu, Fang Zou, Xing Peng, Yiling Lou · 2025 · arXiv
The first empirical study to investigate the prevalence and nature of repetition across 19 state-of-the-art code LLMs using three widely-used benchmarks finds that repetition is pervasive and manifests at various granularities and extents, including character, statement, and block levels.
failure-mode-taxonomybenchmarkempiricalThe Code Copycat Conundrum: Repetition in LLM Code Generation
Mingwei Liu, Yanjun Li, Chong Wang, Xin Du, Tao Ou · 2025 · arXiv
Empirical study of intra-generation repetition in LLM code output. The closest semantic neighbor to SAICA-KG's dependency_blindness class, but studies repetition *within* a single generation rather than cross-codebase reimplementation of existing library code.
repetitionreusedependency-blindness-adjacentAdvancing Multi-Agent Systems Through Model Context Protocol: Architecture, Implementation, and Applications
N. Krishnan · 2025 · arXiv
A comprehensive framework for advancing multi-agent systems through Model Context Protocol (MCP), addressing these challenges through standardized context sharing and coordination mechanisms through standardized context sharing and coordination mechanisms is introduced.
MCPbenchmarkEnhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latency
Nazmus Ashrafi, Salah Bouktif, Mohammed Mediani · 2025 · arXiv
This paper combines two widely used approaches namely multi-agent collaboration and runtime execution information-based debugging, for improving code generation functionality, reliability, and practical applicability and implements a chained system that combines both strategies.
supervisionbenchmarkempiricalMeasuring AI Agent Autonomy: Towards a Scalable Approach with Code Inspection
Peter Cihon, Merlin Stein, Gagan Bansal, Sam Manning, Kevin Xu · 2025 · arXiv
Introduces a code-inspection-based assessment of AI-agent autonomy that does not require running the agent. Orchestration code is scored on two axes — impact and oversight — using a taxonomy demonstrated on AutoGen.
autonomystatic-analysissupervision-vocabularyoversightimpactAutomatic Red Teaming LLM-based Agents with Model Context Protocol Tools
Ping He, Changjiang Li, Binbin Zhao, Tianyu Du, Shouling Ji · 2025 · arXiv
This work proposes AutoMalTool, an automated red teaming framework for LLM-based agents by generating malicious MCP tools capable of manipulating the behavior of mainstream LLM-based agents while evading current detection mechanisms, thereby revealing new security risks in these agents.
MCPbenchmarkred-teamingFrom Glue-Code to Protocols: A Critical Analysis of A2A and MCP Integration for Scalable Agent Systems
Qiaomu Li, Ying Xie · 2025 · arXiv
This work provides a critical analysis, moving beyond a survey to evaluate the practical implications and inherent difficulties of combining these horizontal and vertical integration standards, and examines the benefits (e.g., specialization, scalability) while critically assessing their dependencies and trade-offs in an integrated context.
MCPsurveyWhen the Code Autopilot Breaks: Why LLMs Falter in Embedded Machine Learning
Roberto Morabito, Guanghan Wu · 2025 · arXiv
Empirical study of LLM code-generation failure modes in embedded ML pipelines; derives a failure-category taxonomy emphasizing format-induced misinterpretations and runtime-disruptive-but-compiling code.
failure-mode-taxonomycoding-agentembedded-mlempiricalTrustworthy Agentic AI: Balancing Autonomy with Human Oversight
S. Atri · 2025 · International journal of computer science and mobile computing
Agentic AI systems increasingly plan, decide, and execute actions across tools and data sources with minimal supervision. To be deployable in safety- and compliance-sensitive environments, autonomy must be bounded by transparent decision making, predictable behavior, and recoverable failure modes.
failure-mode-taxonomyagentic-aisupervisionautonomyZero-Trust Security Frameworks for Model Context Protocol (MCP) Server Communications
S. Kota · 2025 · International Journal of Innovative Research in Engineering & Multidisciplinary Physical Sciences
This paper analyzes the threat surface introduced by MCP communications and proposes a structured zero-trust framework tailored to the MCP ecosystem, encompassing identity and device verification, mutual authentication, micro-segmentation, dynamic policy enforcement, continuous monitoring, and breach containment.
agentic-aisupervisionMCPbenchmarkRethinking Autonomy: Preventing Failures in AI-Driven Software Engineering
Satyam Kumar Navneet, Joydeep Chandra · 2025 · arXiv
Analyzes failure classes in LLM-assisted code generation (vulnerability inheritance, overtrust, misinterpretation, absence of validation/rollback) and proposes the SAFE-AI framework (Safety, Auditability, Feedback, Explainability). Introduces a suggestive / generative / autonomous / destructive taxonomy of AI behaviors to guide oversight.
autonomysupervision-vocabularyfailure-mode-taxonomygovernancedestructive-actionMCPToolBench++: A Large Scale AI Agent Model Context Protocol MCP Tool Use Benchmark
Shiqing Fan, Xichen Ding, Liang Zhang, Linjian Mo · 2025 · arXiv
This work proposes MCPToolBench++, a large-scale, multi-domain AI Agent tool use benchmark, build upon marketplace of over 4k MCP servers from more than 40 categories, collected from the MCP marketplaces and GitHub communities.
agentic-aiMCPbenchmarkAn Empirical Study on Failures in Automated Issue Solving
Simiao Liu, Fang Liu, Liehao Li, Xin Tan, Yinghao Zhu, Xiaoli Lian, Li Zhang · 2025 · arXiv
Automated issue solving seeks to autonomously identify and repair defective code snippets across an entire codebase. SWE-Bench has emerged as the most widely adopted benchmark for evaluating progress in this area.
failure-mode-taxonomycoding-agentagentic-aisupervisionbenchmarkempiricalHelp or Hurdle? Rethinking Model Context Protocol-Augmented Large Language Models
Wei Song, Haonan Zhong, Ziqi Ding, Jingling Xue, Yuekang Li · 2025 · arXiv
MCPGAUGE is introduced, the first comprehensive evaluation framework for probing LLM-MCP interactions along four key dimensions: proactivity (self-initiated tool use), compliance, effectiveness, effectiveness, and overhead (computational cost incurred).
MCPbenchmarkWhat to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond
Wenchao Gu, Juntao Chen, Yanlin Wang, Tianyue Jiang, Xing Li, Mingwei Liu, Xilin Liu, Yuchi Ma, Zibin Zheng · 2025 · arXiv
AllianceCoder is proposed, a novel context-integrated method that employs chain-of-thought prompting to decompose user queries into implementation steps and retrieves APIs via semantic description matching and achieves state-of-the-art performance.
benchmarkempiricalMCP-Guard: A Multi-Stage Defense-in-Depth Framework for Securing Model Context Protocol in Agentic AI
Wenpeng Xing, Zhong-Kuo Qi, Yupeng Qin, Yilin Li, Caini Chang, Jiahui Yu, Changting Lin, Zhe Xie, Meng Han · 2025 · arXiv
This work proposes MCP-GUARD, a robust, layered defense architecture designed for LLM-tool interactions, and introduces MCP-ATTACKBENCH, a comprehensive benchmark that simulates diverse real-world attack vectors that circumvent conventional defenses in the MCP paradigm.
agentic-aiMCPbenchmarkSemanticForge: Repository-Level Code Generation through Semantic Knowledge Graphs and Constraint Satisfaction
Wuyang Zhang, Chenkai Zhang, Zhen Luo, Jianming Ma, Wangming Yuan, Chuqiao Gu, Chenwei Feng · 2025 · INNO-PRESS: Journal of Emerging Applied AI
The explicit semantic representation and constraint satisfaction framework introduced in framework enables more reliable automated development tools and provides a foundation for future advances in AI-assisted software engineering.
failure-mode-taxonomybenchmarkknowledge-graphAgentBay: A Hybrid Interaction Sandbox for Seamless Human-AI Intervention in Agentic Systems
Y. Piao, Hongbo Min, Hang Su, Leilei Zhang, Lei Wang, Yue Yin, Xiao Wu, Zhe Xu, Liwei Qu, Hang Li, Xinxin Zeng, Wei Tian, Fei Yu, Xiaowei Li, Jiayi Jiang, Tongxuan Liu, Haoxiang Tian, Yufei Que, Xiao-Qing Tu, Bing Suo, Yuebing Li, Xiang-Hui Chen, Ze-Yu Zhao, Jia-Xiang Tang, Wei Huang, Xuguang Li, Jing Zhao, Jin Li, Jie Shen, Jinkui Ren, Xiantao Zhang · 2025 · arXiv
It is proposed that AgentBay provides a foundational primitive for building the next generation of reliable, human-supervised autonomous systems, especially under poor network conditions.
agentic-aisupervisionMCPbenchmarkguardrailsVisCoder2: Building Multi-Language Visualization Coding Agents
Yuansheng Ni, Songcheng Cai, Xiang Chen, Jiarong Liang, Z. Lyu, Jiaqi Deng, Kai Zou, Ping Nie, Fei Yuan, Xiang Yue, Wenhu Chen · 2025 · arXiv
Experiments show that VisCoder2 significantly outperforms strong open-source baselines and approaches the performance of proprietary models like GPT-4.1, with further gains from iterative self-debug, reaching 82.4% overall execution pass rate at the 32B scale, particularly in symbolic or compiler-dependent languages.
coding-agentsupervisionbenchmarkHFuzzer: Testing Large Language Models for Package Hallucinations via Phrase-based Fuzzing
Yukai Zhao, Menghan Wu, Xing Hu, Xin Xia · 2025 · International Conference on Automated Software Engineering
H HFuzzer is proposed, a novel phrase-based fuzzing framework to test LLMs for package hallucinations that adopts fuzzing technology and guides the model to infer a wider range of reasonable information based on phrases, thereby generating enough and diverse coding tasks.
supply-chainHallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges
Yunseo Lee, John Youngeun Song, Dongsun Kim, Jindae Kim, Mijung Kim, Jaechang Nam · 2025 · arXiv
Survey categorizing hallucination types in Code-LLM output, reviewing existing benchmarks and mitigation strategies, and identifying open challenges in detecting and removing hallucinated code.
failure-mode-taxonomyhallucinationsurveycode-generationWhere LLM Agents Fail: A Taxonomy of Errors and the AgentErrorBench Dataset
Zhu, et al. · 2025 · arXiv
AgentErrorTaxonomy organizes LLM agent failures by module (memory, reflection, planning, action, system). Paired with AgentErrorBench dataset.
failure-mode-taxonomyagent-errorsMCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers
Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, Junnan Li · 2025 · arXiv
First comprehensive benchmark of LLM agents against real-world MCP servers across 6 domains and 11 MCP servers. Uses execution-based evaluators (format, static, dynamic). Even GPT-5 reaches only 43.72%; exposes the long-context and unknown-tools challenges inherent to MCP.
benchmarkMCPagent-toolreal-world-evaluationsupply-chain-adjacentLLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion
Chong Wang, Kaifeng Huang, Jian Zhang, et al. · 2024 · ICSE
Canonical study of deprecated-API usage in LLM-based code completion; shows systematic tendency of LLMs to use retired API versions.
obsolescencedeprecated-apifailure-mode-evidenceBeyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
Fang Liu, Yang Liu, Lin Shi, Houkun Yang, Zhen Zhang · 2024 · IEEE Transactions on Software Engineering
Empirical study and taxonomy of hallucinations in LLM-generated code, with 3/12 category classification that is the closest prior art to SAICA-KG's FailureMode axis. Conflates axes (cause, symptom, type) rather than partitioning them orthogonally.
failure-mode-taxonomyhallucinationcode-generationWe Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs
Joseph Spracklen, Raveen Wijewickrama, Nazmus Sakib, Anindya Maiti, Bimal Viswanath · 2024 · USENIX Security
Canonical empirical study of "slopsquatting" — package hallucinations by code-generating LLMs. Defines the phenomenon and measures prevalence across commercial and open-source models.
supply-chainpackage-hallucinationslopsquattingfailure-mode-evidenceA Survey on Large Language Models for Code Generation
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, Sunghun Kim · 2024 · ACM Transactions on Software Engineering and Methodology (TOSEM)
Systematic literature review of LLMs for code generation, organizing the field around data curation, advances, evaluation, ethics, environment, and real-world applications. Introduces a taxonomy of Code LLM developments and compares HumanEval/MBPP/BigCodeBench results across models.
surveycode-generationlandscapebackground-citationCodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
Yuchen Tian, Weixiang Yan, Qian Yang, Xuandong Zhao, Qian Chen, Wen Wang, Ziyang Luo, Lei Ma · 2024 · AAAI Conference on Artificial Intelligence
Introduces the concept of code hallucinations and a four-class execution- verification taxonomy (mapping, naming, resource, logic). Provides the CodeHalu detection algorithm and the CodeHaluEval benchmark (8,883 samples over 699 tasks) across 17 LLMs.
failure-mode-taxonomyhallucinationcode-generationbenchmarkexecution-based-verificationLLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation
Ziyao Zhang, Chong Wang, Yanlin Wang, Ensheng Shi, Yuchi Ma, Wanjun Zhong, Jiachi Chen, Mingzhi Mao, Zibin Zheng · 2024 · Proceedings of the ACM on Software Engineering (PACMSE / FSE)
Empirical study of hallucinations in repository-level code generation across six mainstream LLMs. Derives a hallucination taxonomy, distributes phenomena by model, identifies four contributing factors, and proposes an RAG-based mitigation that is effective across the studied models.
failure-mode-taxonomyhallucinationrepository-levelcode-generationmitigationTaxonomies in Software Engineering: A Systematic Mapping Study and a Revised Taxonomy Development Method
Muhammad Usman, Ricardo Britto, Jürgen Börstler, Emilia Mendes · 2017 · Information and Software Technology
Canonical mapping study on taxonomy construction methods in SE. Grounds the methodological spine that SAICA-KG's honest-MECE approach inherits.
methodologytaxonomy-constructionfaceted-classification