[
  {
    "id": "ashrafi-2025-enhancing-llm-code-generation",
    "authors": [
      "Nazmus Ashrafi",
      "Salah Bouktif",
      "Mohammed Mediani"
    ],
    "year": 2025,
    "title": "Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latency",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/92b2ea3c244855235d10ca270dea7348fae07a25",
    "doi": "10.48550/arXiv.2505.02133",
    "arxiv_id": "2505.02133",
    "semantic_scholar_id": "92b2ea3c244855235d10ca270dea7348fae07a25",
    "abstract": "The use of large language models (LLMs) for automated code generation has emerged as a significant focus within AI research. As these pretrained models continue to evolve, their ability to understand and generate complex code structures has opened new possibilities for automating intricate programming tasks for the sake of accurate code generation. Although contemporary foundational models demonstrate promoting results, researchers continue to explore optimal post-training strategies to enhance code quality. These include supervised fine-tuning, retrieval-augmented generation (RAG), debugging, and many others. In this paper, we combine two widely used approaches namely multi-agent collaboration and runtime execution information-based debugging, for improving code generation functionality, reliability, and practical applicability. We perform an empirical study in order to extend the evaluation of the individual strategies as well as the proposed composition of the activities of both strategies. Our study use 19 LLMs to examines the performance of individual and the proposed strategies, offering comprehensive insights into how different programming activities compositions and training paradigms influence code generation effectiveness. In particular, we implement a chained system that combines both strategies to assess their combined impact on functional accuracy, code reliability, and generation latency using two benchmark datasets commonly used for code generation. Our findings provide valuable insights for organizations seeking robust AI-driven coding solutions by guiding them in selecting models that can better adapt to complex post-training strategies, ultimately fostering the adoption of more effective and reliable code generation technologies.\n",
    "tldr": "This paper combines two widely used approaches namely multi-agent collaboration and runtime execution information-based debugging, for improving code generation functionality, reliability, and practical applicability and implements a chained system that combines both strategies.\n",
    "citation_count": 12,
    "relevance_tags": [
      "supervision",
      "benchmark",
      "empirical"
    ],
    "notes": "Empirical study chaining multi-agent collaboration with runtime-execution-information-based debugging across 19 LLMs on HumanEval and MBPP, measuring functional accuracy, reliability, and generation latency. Useful as evidence that runtime feedback loops (a post-generation supervision pattern) reduce logic_error rates over single-shot generation, but also as evidence that naive chaining inflates latency. Relates to SAICA-KG failure modes: logic_error, incomplete_execution. Informs the post_generation TemporalPhase axis: runtime-debug agents act as a self-correcting layer that shifts where supervision catches faults.\n"
  },
  {
    "id": "atri-2025-trustworthy-agentic-ai-balancing",
    "authors": [
      "S. Atri"
    ],
    "year": 2025,
    "title": "Trustworthy Agentic AI: Balancing Autonomy with Human Oversight",
    "venue": "International journal of computer science and mobile computing",
    "url": "https://doi.org/10.47760/ijcsmc.2025.v14i11.013",
    "doi": "10.47760/ijcsmc.2025.v14i11.013",
    "semantic_scholar_id": "284745809",
    "abstract": "Agentic AI systems increasingly plan, decide, and execute actions across tools and data sources with minimal supervision. To be deployable in safety- and compliance-sensitive environments, autonomy must be bounded by transparent decision making, predictable behavior, and recoverable failure modes. This paper proposes a practical pattern for trustworthy agentic AI that integrates a policy gate, typed tools and effects, human-in-the-loop (HITL) controls, layered safety monitors, and rigorous verification via evidence logs. We align these choices with the NIST AI Risk Management Framework and its Generative AI Profile, ISO/IEC 42001 for AI management systems, and the staged implementation of the EU AI Act [1–6].\n",
    "tldr": "Agentic AI systems increasingly plan, decide, and execute actions across tools and data sources with minimal supervision. To be deployable in safety- and compliance-sensitive environments, autonomy must be bounded by transparent decision making, predictable behavior, and recoverable failure modes.\n",
    "relevance_tags": [
      "failure-mode-taxonomy",
      "agentic-ai",
      "supervision",
      "autonomy"
    ],
    "notes": "Pattern paper proposing a trustworthy-agentic-AI architecture built from a policy gate, typed tools and effects, human-in-the-loop (HITL) controls, layered safety monitors, and evidence logs, aligned to NIST AI RMF, ISO/IEC 42001, and the EU AI Act. Light on empirical content but useful as a supervision-vocabulary anchor for SAICA-KG's ControlParadigm (policy gate, HITL) and TemporalPhase (real-time monitor vs. ex-post evidence log) partitions. Relates to SAICA-KG failure modes: scope_creep, security_vulnerability, incomplete_execution. Cite for the typed-effects framing when arguing that supervision must be explicit at the tool boundary.\n"
  },
  {
    "id": "bhattarai-2025-arcs-agentic-retrieval-augmented",
    "authors": [
      "Manish Bhattarai",
      "Miguel Cordova",
      "Javier E. Santos",
      "Dan O'Malley"
    ],
    "year": 2025,
    "title": "ARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative Refinement",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/172e194d8b190377e12ffad78d025ca32459cb79",
    "doi": "10.48550/arXiv.2504.20434",
    "arxiv_id": "2504.20434",
    "semantic_scholar_id": "172e194d8b190377e12ffad78d025ca32459cb79",
    "abstract": "We present Agentic Retrieval-Augmented Code Synthesis (ARCS), a system that improves LLM-based code generation without fine-tuning. ARCS operates through a budgeted synthesize-execute-repair loop over a frozen model: it retrieves relevant code context before generation, proposes candidates, executes them against tests, and repairs based on execution feedback. This retrieval-before-generation design reduces hallucination and accelerates convergence. We formalize ARCS as a state-action process with provable guarantees on termination, monotonic improvement, and bounded cost. A tiered controller (Small/Medium/Large) trades latency for accuracy predictably. On HumanEval, ARCS achieves up to 87.2% pass@1 with Llama-3.1-405B, surpassing CodeAgent (82.3%) while using simpler control than tree-search methods. On TransCoder, it achieves>= 90% accuracy on most translation pairs. On a LANL scientific corpus, it improves CodeBLEU by +0.115 over baseline RAG. ARCS provides a practical, reproducible approach to reliable code synthesis using existing LLM checkpoints.\n",
    "tldr": "ARS provides a practical, reproducible approach to reliable code synthesis using existing LLM checkpoints using existing LLM checkpoints as a state-action process with provable guarantees on termination, monotonic improvement, and bounded cost.\n",
    "citation_count": 7,
    "relevance_tags": [
      "agentic-ai"
    ],
    "notes": "Introduces ARCS (Agentic Retrieval-Augmented Code Synthesis), a budgeted synthesize-execute-repair loop over a frozen LLM with formal guarantees on termination, monotonic improvement, and bounded cost. Evaluated on HumanEval, TransCoder, and a LANL scientific corpus; reports 87.2% pass@1 on HumanEval with Llama-3.1-405B vs. CodeAgent at 82.3%. The retrieval-before-generation design is a concrete example of a pre_generation guardrail that targets fabrication. Relates to SAICA-KG failure modes: fabrication, logic_error. Useful as a case study for the ControlParadigm axis: a tiered controller (Small/Medium/Large) is a worked example of cost-aware supervision.\n"
  },
  {
    "id": "bulut-2026-avda-autonomous-vibe-detection",
    "authors": [
      "Fatih Bulut",
      "C. Depaolis",
      "Raghav Batta",
      "Anjali Mangal"
    ],
    "year": 2026,
    "title": "AVDA: Autonomous Vibe Detection Authoring for Cybersecurity",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/a4343100a053fd94c99800cd388de3f5d17fd612",
    "doi": "10.1145/3803437.3805261 10.1145/3803437.3805261 10.1145/3803437.3805261",
    "arxiv_id": "2603.25930",
    "semantic_scholar_id": "a4343100a053fd94c99800cd388de3f5d17fd612",
    "abstract": "With the rapid advancement of AI in code generation, cybersecurity detection engineering faces new opportunities to automate traditionally manual processes. Detection authoring - the practice of creating executable logic that identifies malicious activities from security telemetry - is hindered by fragmented code across repositories, duplication, and limited organizational visibility. Current workflows remain heavily manual, constraining both coverage and velocity. In this paper, we introduce AVDA, a framework that leverages the Model Context Protocol (MCP) to automate detection authoring by integrating organizational context - existing detections, telemetry schemas, and style guides - into AI-assisted code generation. We evaluate three authoring strategies - Baseline, Sequential, and Agentic - across a diverse corpus of production detections and state-of-the-art LLMs. Our results show that Agentic workflows achieve a 19% improvement in overall similarity score over Baseline approaches, while Sequential workflows attain 87% of Agentic quality at 40x lower token cost. Generated detections excel at TTP matching (99.4%) and syntax validity (95.9%) but struggle with exclusion parity (8.9%). Expert validation on a 22-detection subset confirms strong Spearman correlation between automated metrics and practitioner judgment ($\\rho = 0.64$, $p<0.002$). By integrating seamlessly into standard developer environments, AVDA provides a practical path toward AI-assisted detection engineering with quantified trade-offs between quality, cost, and latency.\n",
    "tldr": "AVDA is introduced, a framework that leverages the Model Context Protocol to automate detection authoring by integrating organizational context - existing detections, telemetry schemas, and style guides - into AI-assisted code generation by integrating seamlessly into standard developer environments.\n",
    "relevance_tags": [
      "agentic-ai",
      "MCP"
    ],
    "notes": "Introduces AVDA, an MCP-based framework for automating cybersecurity detection authoring by feeding organizational context (existing detections, telemetry schemas, style guides) into AI-assisted code generation. Compares Baseline, Sequential, and Agentic workflows; reports a 19% similarity uplift for Agentic over Baseline at 40x higher token cost than Sequential. Notable failure pattern: 8.9% exclusion-parity score (the agent struggles to honor explicit not-this-detection constraints) — a clear instance of scope_creep when the agent is pointed at security telemetry. Relates to SAICA-KG failure modes: scope_creep, context_pollution. Useful as MCP integration evidence and as a worked example of how organizational context becomes supervision substrate.\n"
  },
  {
    "id": "cemri-2025-mast",
    "authors": [
      "Melis Cemri",
      "et al."
    ],
    "year": 2025,
    "title": "Why Do Multi-Agent LLM Systems Fail? MAST: A Multi-Agent System Failure Taxonomy",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2503.13657",
    "arxiv_id": "2503.13657",
    "abstract": "Despite enthusiasm for Multi-Agent LLM Systems (MAS), their performance gains on popular benchmarks are often minimal. This gap highlights a critical need for a principled understanding of why MAS fail. Addressing this question requires systematic identification and analysis of failure patterns. We introduce MAST-Data, a comprehensive dataset of 1600+ annotated traces collected across 7 popular MAS frameworks. MAST-Data is the first multi-agent system dataset to outline the failure dynamics in MAS for guiding the development of better future systems. To enable systematic classification of failures for MAST-Data, we build the first Multi-Agent System Failure Taxonomy (MAST). We develop MAST through rigorous analysis of 150 traces, guided closely by expert human annotators and validated by high inter-annotator agreement (kappa = 0.88). This process identifies 14 unique modes, clustered into 3 categories: (i) system design issues, (ii) inter-agent misalignment, and (iii) task verification. To enable scalable annotation, we develop an LLM-as-a-Judge pipeline with high agreement with human annotations. We leverage MAST and MAST-Data to analyze failure patterns across models (GPT4, Claude 3, Qwen2.5, CodeLlama) and tasks (coding, math, general agent), demonstrating improvement headrooms from better MAS design. Our analysis provides insights revealing that identified failures require more sophisticated solutions, highlighting a clear roadmap for future research. We publicly release our comprehensive dataset (MAST-Data), the MAST, and our LLM annotator to facilitate widespread research and development in MAS.",
    "tldr": "Grounded-theory taxonomy of 14 multi-agent failure modes across 3 clusters, derived from 1,600 annotated traces over 7 MAS frameworks.\n",
    "relevance_tags": [
      "failure-mode-taxonomy",
      "multi-agent",
      "external-taxonomy-seed"
    ],
    "notes": "Must-cite. Seeds the MAST Taxonomy node in SAICA-KG and is a primary input for the cross-taxonomy absorption contribution.\n"
  },
  {
    "id": "cihon-stein-2025-autonomy-scoring",
    "authors": [
      "Peter Cihon",
      "Merlin Stein",
      "Gagan Bansal",
      "Sam Manning",
      "Kevin Xu"
    ],
    "year": 2025,
    "title": "Measuring AI Agent Autonomy: Towards a Scalable Approach with Code Inspection",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2502.15212",
    "doi": "10.48550/arXiv.2502.15212",
    "arxiv_id": "2502.15212",
    "semantic_scholar_id": "783c566fc06acd6b10a6679c2363d5d0cc26f0c2",
    "abstract": "AI agents are AI systems that can achieve complex goals autonomously. Assessing the level of agent autonomy is crucial for understanding both their potential benefits and risks. Current assessments of autonomy often focus on specific risks and rely on run-time evaluations -- observations of agent actions during operation. We introduce a code-based assessment of autonomy that eliminates the need to run an AI agent to perform specific tasks, thereby reducing the costs and risks associated with run-time evaluations. Using this code-based framework, the orchestration code used to run an AI agent can be scored according to a taxonomy that assesses attributes of autonomy: impact and oversight. We demonstrate this approach with the AutoGen framework and select applications.",
    "tldr": "Introduces a code-inspection-based assessment of AI-agent autonomy that does not require running the agent. Orchestration code is scored on two axes — impact and oversight — using a taxonomy demonstrated on AutoGen.\n",
    "citation_count": 11,
    "relevance_tags": [
      "autonomy",
      "static-analysis",
      "supervision-vocabulary",
      "oversight",
      "impact"
    ],
    "notes": "Must-cite. Direct external anchor for SAICA-KG's AutonomyLevel partition — see enums.yml where AutonomyLevel values are explicitly aligned to Cihon-Stein 2025 (high impact + low oversight -> fully_autonomous; etc.). Cite to neutralize \"your autonomy axis is arbitrary\" critiques.\n"
  },
  {
    "id": "ehsani-2026-where-ai-agents-fail",
    "authors": [
      "Ramtin Ehsani",
      "Sakshi Pathak",
      "Sarthak Rawal",
      "Abdullah Mujahid",
      "M. M. Imran",
      "Preetha Chatterjee"
    ],
    "year": 2026,
    "title": "Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2601.15195",
    "doi": "10.48550/arXiv.2601.15195",
    "arxiv_id": "2601.15195",
    "semantic_scholar_id": "fc9ea2db0899a017c9bd5fd9656a4c12b87ae288",
    "abstract": "AI coding agents are now submitting pull requests (PRs) to software projects, acting not just as assistants but as autonomous contributors. As these agentic contributions are rapidly increasing across real repositories, little is known about how they behave in practice and why many of them fail to be merged. In this paper, we conduct a large-scale study of 33k agent-authored PRs made by five coding agents across GitHub. (RQ1) We first quantitatively characterize merged and not-merged PRs along four broad dimensions: 1) merge outcomes across task types, 2) code changes, 3) CI build results, and 4) review dynamics. We observe that tasks related to documentation, CI, and build update achieve the highest merge success, whereas performance and bug-fix tasks perform the worst. Not-merged PRs tend to involve larger code changes, touch more files, and often do not pass the project's CI/CD pipeline validation. (RQ2) To further investigate why some agentic PRs are not merged, we qualitatively analyze 600 PRs to derive a hierarchical taxonomy of rejection patterns. This analysis complements the quantitative findings in RQ1 by uncovering rejection reasons not captured by quantitative metrics, including lack of meaningful reviewer engagement, duplicate PRs, unwanted feature implementations, and agent misalignment. Together, our findings highlight key socio-technical and human-AI collaboration factors that are critical to improving the success of future agentic workflows.",
    "tldr": "Large-scale study of 33k agent-authored PRs across five coding agents on GitHub. Quantifies merge outcomes by task type, code-change size, CI, and review dynamics; derives a hierarchical taxonomy of rejection patterns from qualitative analysis of 600 PRs (duplicate PRs, unwanted features, lack of reviewer engagement, agent misalignment).\n",
    "citation_count": 2,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "coding-agent",
      "github",
      "empirical",
      "agent-misalignment"
    ],
    "notes": "Must-cite. Single best empirical study of *deployed* coding-agent failure signatures. Grounds SAICA-KG's scope_creep and context_pollution FailureModes with real PR-rejection data and validates the KG's multi-agent-PR framing.\n"
  },
  {
    "id": "fan-2025-mcptoolbench-large-scale-ai",
    "authors": [
      "Shiqing Fan",
      "Xichen Ding",
      "Liang Zhang",
      "Linjian Mo"
    ],
    "year": 2025,
    "title": "MCPToolBench++: A Large Scale AI Agent Model Context Protocol MCP Tool Use Benchmark",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/153e3227cdc8e8b54034b6166a468bd751e117cc",
    "doi": "10.48550/arXiv.2508.07575",
    "arxiv_id": "2508.07575",
    "semantic_scholar_id": "153e3227cdc8e8b54034b6166a468bd751e117cc",
    "abstract": "LLMs'capabilities are enhanced by using function calls to integrate various data sources or API results into the context window. Typical tools include search, web crawlers, maps, financial data, file systems, and browser usage, etc. Integrating these data sources or functions requires a standardized method. The Model Context Protocol (MCP) provides a standardized way to supply context to LLMs. However, the evaluation of LLMs and AI Agents'MCP tool use abilities suffer from several issues. First, there's a lack of comprehensive datasets or benchmarks to evaluate various MCP tools. Second, the diverse formats of response from MCP tool call execution further increase the difficulty of evaluation. Additionally, unlike existing tool-use benchmarks with high success rates in functions like programming and math functions, the success rate of real-world MCP tool is not guaranteed and varies across different MCP servers. Furthermore, the LLMs'context window also limits the number of available tools that can be called in a single run, because the textual descriptions of tool and the parameters have long token length for an LLM to process all at once. To help address the challenges of evaluating LLMs'performance on calling MCP tools, we propose MCPToolBench++, a large-scale, multi-domain AI Agent tool use benchmark. As of July 2025, this benchmark is build upon marketplace of over 4k MCP servers from more than 40 categories, collected from the MCP marketplaces and GitHub communities. The datasets consist of both single-step and multi-step tool calls across different categories. We evaluated SOTA LLMs with agentic abilities on this benchmark and reported the results.\n",
    "tldr": "This work proposes MCPToolBench++, a large-scale, multi-domain AI Agent tool use benchmark, build upon marketplace of over 4k MCP servers from more than 40 categories, collected from the MCP marketplaces and GitHub communities.\n",
    "citation_count": 23,
    "relevance_tags": [
      "agentic-ai",
      "MCP",
      "benchmark"
    ],
    "notes": "Introduces MCPToolBench++, a large-scale multi-domain MCP tool use benchmark built on a marketplace of over 4k MCP servers across 40+ categories collected from MCP marketplaces and GitHub. Datasets include single-step and multi-step tool calls evaluated against SOTA agentic LLMs. Notable for explicitly calling out two SAICA-relevant pressures: (a) real-world MCP server success rates are not guaranteed and vary across servers (incomplete_execution), and (b) the LLM context window caps the number of available tools that can be called in a single run because tool descriptions are long-token (context_pollution). Relates to SAICA-KG failure modes: context_pollution, incomplete_execution. Cite as the canonical MCP tool-use evaluation harness.\n"
  },
  {
    "id": "gao-2026-skillreducer-optimizing-llm-agent",
    "authors": [
      "Yudong Gao",
      "Zongjie Li",
      "Yuanyuanyuan",
      "Zimo Ji",
      "Pingchuan Ma",
      "Shuai Wang"
    ],
    "year": 2026,
    "title": "SkillReducer: Optimizing LLM Agent Skills for Token Efficiency",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/515bbd3bda703bdf891d373713451f985f6a0bc9",
    "arxiv_id": "2603.29919",
    "semantic_scholar_id": "515bbd3bda703bdf891d373713451f985f6a0bc9",
    "abstract": "LLM-based coding agents rely on \\emph{skills}, pre-packaged instruction sets that extend agent capabilities, yet every token of skill content injected into the context window incurs both monetary cost and attention dilution. To understand the severity of this problem, we conduct a large-scale empirical study of 55,315 publicly available skills and find systemic inefficiencies: 26.4\\% lack routing descriptions entirely, over 60\\% of body content is non-actionable, and reference files can inject tens of thousands of tokens per invocation. Motivated by these findings, we present \\textsc{SkillReducer}, a two-stage optimization framework. Stage~1 optimizes the routing layer by compressing verbose descriptions and generating missing ones via adversarial delta debugging. Stage~2 restructures skill bodies through taxonomy-driven classification and progressive disclosure, separating actionable core rules from supplementary content loaded on demand, validated by faithfulness checks and a self-correcting feedback loop. Evaluated on 600 skills and the SkillsBench benchmark, \\textsc{SkillReducer} achieves 48\\% description compression and 39\\% body compression while improving functional quality by 2.8\\%, revealing a \\emph{less-is-more} effect where removing non-essential content reduces distraction in the context window. These benefits transfer across five models from four families with a mean retention of 0.965, and generalize to an independent agent framework.\n",
    "tldr": "A two-stage optimization framework that achieves a less-is-more effect where removing non-essential content reduces distraction in the context window, and generalize to an independent agent framework is presented.\n",
    "relevance_tags": [
      "failure-mode-taxonomy",
      "coding-agent",
      "agentic-ai",
      "benchmark",
      "empirical"
    ],
    "notes": "Large-scale empirical study of 55,315 publicly available agent skills (pre-packaged instruction sets) finding systemic inefficiencies: 26.4% lack routing descriptions, over 60% of body content is non-actionable, and reference files inject tens of thousands of tokens per invocation. Proposes SkillReducer, a two-stage compression framework evaluated on 600 skills and the SkillsBench benchmark, achieving 48% description and 39% body compression while improving functional quality by 2.8%. The 'less-is-more' finding directly evidences context_pollution as a supervision-relevant failure mode. Relates to SAICA-KG failure modes: context_pollution, scope_creep. Useful for arguing that context curation is a first-class supervision activity, not just a cost optimization.\n"
  },
  {
    "id": "gu-2025-retrieve-effective-retrieval-augmented",
    "authors": [
      "Wenchao Gu",
      "Juntao Chen",
      "Yanlin Wang",
      "Tianyue Jiang",
      "Xing Li",
      "Mingwei Liu",
      "Xilin Liu",
      "Yuchi Ma",
      "Zibin Zheng"
    ],
    "year": 2025,
    "title": "What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/695f334494f305d2ff57f2d584c27cdcaa4012ff",
    "doi": "10.48550/arXiv.2503.20589",
    "arxiv_id": "2503.20589",
    "semantic_scholar_id": "695f334494f305d2ff57f2d584c27cdcaa4012ff",
    "abstract": "Repository-level code generation remains challenging due to complex code dependencies and the limitations of large language models (LLMs) in processing long contexts. While retrieval-augmented generation (RAG) frameworks are widely adopted, the effectiveness of different retrieved information sources-contextual code, APIs, and similar snippets-has not been rigorously analyzed. Through an empirical study on two benchmarks, we demonstrate that in-context code and potential API information significantly enhance LLM performance, whereas retrieved similar code often introduces noise, degrading results by up to 15%. Based on the preliminary results, we propose AllianceCoder, a novel context-integrated method that employs chain-of-thought prompting to decompose user queries into implementation steps and retrieves APIs via semantic description matching. Through extensive experiments on CoderEval and RepoExec, AllianceCoder achieves state-of-the-art performance, improving Pass@1 by up to 20% over existing approaches.\n",
    "tldr": "AllianceCoder is proposed, a novel context-integrated method that employs chain-of-thought prompting to decompose user queries into implementation steps and retrieves APIs via semantic description matching and achieves state-of-the-art performance.\n",
    "citation_count": 11,
    "relevance_tags": [
      "benchmark",
      "empirical"
    ],
    "notes": "Empirical study of which retrieved-information sources actually help repository-level code generation. Key finding: in-context code and potential API information help, but retrieved similar-code snippets often introduce noise that degrades results by up to 15%. Proposes AllianceCoder with chain-of-thought decomposition and semantic-description API retrieval; evaluated on CoderEval and RepoExec with up to 20% Pass@1 improvement. Direct evidence that uncurated RAG context is a context_pollution vector. Relates to SAICA-KG failure modes: context_pollution, dependency_blindness. Cite when arguing that retrieval-quality is itself a supervisable surface.\n"
  },
  {
    "id": "haque-2025-secure-suspect-investigating-package",
    "authors": [
      "Md Nazmul Haque",
      "Elizabeth Lin",
      "L. Arkoh",
      "B. Tadesse",
      "Bowen Xu"
    ],
    "year": 2025,
    "title": "Secure or Suspect? Investigating Package Hallucinations of Shell Command in Original and Quantized LLMs",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/91cf61ed35840fb8676d5cda64c55267e5a929ae",
    "doi": "10.48550/arXiv.2512.08213",
    "arxiv_id": "2512.08213",
    "semantic_scholar_id": "91cf61ed35840fb8676d5cda64c55267e5a929ae",
    "abstract": "Large Language Models for code (LLMs4Code) are increasingly used to generate software artifacts, including library and package recommendations in languages such as Go. However, recent evidence shows that LLMs frequently hallucinate package names or generate dependencies containing known security vulnerabilities, posing significant risks to developers and downstream software supply chains. At the same time, quantization has become a widely adopted technique to reduce inference cost and enable deployment of LLMs on resource-constrained environments. Despite its popularity, little is known about how quantization affects the correctness and security of LLM-generated software dependencies while generating shell commands for package installation. In this work, we conduct the first systematic empirical study of the impact of quantization on package hallucination and vulnerability risks in LLM-generated Go packages. We evaluate five Qwen model sizes under full-precision, 8-bit, and 4-bit quantization across three datasets (SO, MBPP, and paraphrase). Our results show that quantization substantially increases the package hallucination rate (PHR), with 4-bit models exhibiting the most severe degradation. We further find that even among the correctly generated packages, the vulnerability presence rate (VPR) rises as precision decreases, indicating elevated security risk in lower-precision models. Finally, our analysis of hallucinated outputs reveals that most fabricated packages resemble realistic URL-based Go module paths, such as most commonly malformed or non-existent GitHub and golang.org repositories, highlighting a systematic pattern in how LLMs hallucinate dependencies. Overall, our findings provide actionable insights into the reliability and security implications of deploying quantized LLMs for code generation and dependency recommendation.\n",
    "tldr": "This work conducts the first systematic empirical study of the impact of quantization on package hallucination and vulnerability risks in LLM-generated Go packages and reveals that most fabricated packages resemble realistic URL-based Go module paths, highlighting a systematic pattern in how LLMs hallucinate dependencies.\n",
    "relevance_tags": [
      "supply-chain",
      "empirical"
    ],
    "notes": "First systematic empirical study of how quantization (8-bit and 4-bit vs. full-precision) affects package hallucination and vulnerability rates in LLM-generated Go install commands. Evaluates five Qwen model sizes across SO, MBPP, and paraphrase datasets; finds quantization substantially increases the package hallucination rate (PHR), with 4-bit degrading most, and that vulnerability presence rate (VPR) among correctly-generated packages also rises as precision decreases. Most fabricated packages resemble realistic URL-based Go module paths (malformed GitHub or golang.org repositories) — a slopsquatting-adjacent supply-chain risk. Relates to SAICA-KG failure modes: supply_chain_attack, fabrication, dependency_blindness, security_vulnerability. Cite as evidence that deployment-time decisions (quantization) materially shift FailureMode distributions.\n"
  },
  {
    "id": "he-2025-automatic-red-teaming-llm",
    "authors": [
      "Ping He",
      "Changjiang Li",
      "Binbin Zhao",
      "Tianyu Du",
      "Shouling Ji"
    ],
    "year": 2025,
    "title": "Automatic Red Teaming LLM-based Agents with Model Context Protocol Tools",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/299ec08b60dbb993f96cf45c011c387477e3f9c8",
    "doi": "10.48550/arXiv.2509.21011",
    "arxiv_id": "2509.21011",
    "semantic_scholar_id": "299ec08b60dbb993f96cf45c011c387477e3f9c8",
    "abstract": "The remarkable capability of large language models (LLMs) has led to the wide application of LLM-based agents in various domains. To standardize interactions between LLM-based agents and their environments, model context protocol (MCP) tools have become the de facto standard and are now widely integrated into these agents. However, the incorporation of MCP tools introduces the risk of tool poisoning attacks, which can manipulate the behavior of LLM-based agents. Although previous studies have identified such vulnerabilities, their red teaming approaches have largely remained at the proof-of-concept stage, leaving the automatic and systematic red teaming of LLM-based agents under the MCP tool poisoning paradigm an open question. To bridge this gap, we propose AutoMalTool, an automated red teaming framework for LLM-based agents by generating malicious MCP tools. Our extensive evaluation shows that AutoMalTool effectively generates malicious MCP tools capable of manipulating the behavior of mainstream LLM-based agents while evading current detection mechanisms, thereby revealing new security risks in these agents.\n",
    "tldr": "This work proposes AutoMalTool, an automated red teaming framework for LLM-based agents by generating malicious MCP tools capable of manipulating the behavior of mainstream LLM-based agents while evading current detection mechanisms, thereby revealing new security risks in these agents.\n",
    "citation_count": 10,
    "relevance_tags": [
      "MCP",
      "benchmark",
      "red-teaming"
    ],
    "notes": "Proposes AutoMalTool, an automated red-teaming framework that generates malicious MCP tools capable of manipulating the behavior of mainstream LLM-based agents while evading current detection mechanisms. The first systematic (rather than proof-of-concept) treatment of MCP tool-poisoning attacks. Direct evidence that the MCP integration surface is a supply_chain_attack vector and that detection-mechanism gaps are exploitable. Relates to SAICA-KG failure modes: supply_chain_attack, security_vulnerability. Cite alongside spracklen-2024 and the MCP-Guard / MSB benchmarks when arguing that supply-chain awareness must extend to MCP tool marketplaces, not just package registries.\n"
  },
  {
    "id": "horikawa-2025-agentic-refactoring-empirical-study",
    "authors": [
      "Kosei Horikawa",
      "Hao Li",
      "Yutaro Kashiwa",
      "Bram Adams",
      "Hajimu Iida",
      "Ahmed E. Hassan"
    ],
    "year": 2025,
    "title": "Agentic Refactoring: An Empirical Study of AI Coding Agents",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/65b7e09b4b19bb763f1e1d7b17a4e927e0d8179b",
    "doi": "10.48550/arXiv.2511.04824",
    "arxiv_id": "2511.04824",
    "semantic_scholar_id": "65b7e09b4b19bb763f1e1d7b17a4e927e0d8179b",
    "abstract": "Agentic coding tools, such as OpenAI Codex, Claude Code, and Cursor, are transforming the software engineering landscape. These AI-powered systems function as autonomous teammates capable of planning and executing complex development tasks. Agents have become active participants in refactoring, a cornerstone of sustainable software development aimed at improving internal code quality without altering observable behavior. Despite their increasing adoption, there is a critical lack of empirical understanding regarding how agentic refactoring is utilized in practice, how it compares to human-driven refactoring, and what impact it has on code quality. To address this empirical gap, we present a large-scale study of AI agent-generated refactorings in real-world open-source Java projects, analyzing 15,451 refactoring instances across 12,256 pull requests and 14,988 commits derived from the AIDev dataset. Our empirical analysis shows that refactoring is a common and intentional activity in this development paradigm, with agents explicitly targeting refactoring in 26.1% of commits. Analysis of refactoring types reveals that agentic efforts are dominated by low-level, consistency-oriented edits, such as Change Variable Type (11.8%), Rename Parameter (10.4%), and Rename Variable (8.5%), reflecting a preference for localized improvements over the high-level design changes common in human refactoring. Additionally, the motivations behind agentic refactoring focus overwhelmingly on internal quality concerns, with maintainability (52.5%) and readability (28.1%). Furthermore, quantitative evaluation of code quality metrics shows that agentic refactoring yields small but statistically significant improvements in structural metrics, particularly for medium-level changes, reducing class size and complexity (e.g., Class LOC median $\\Delta$ = -15.25).\n",
    "tldr": "A large-scale study of AI agent-generated refactorings in real-world open-source Java projects, analyzing 15,451 refactoring instances across 12,256 pull requests and 14,988 commits derived from the AIDev dataset shows that refactoring is a common and intentional activity in this development paradigm.\n",
    "citation_count": 5,
    "relevance_tags": [
      "coding-agent",
      "agentic-ai",
      "benchmark",
      "empirical"
    ],
    "notes": "Large-scale empirical study of agentic refactoring on the AIDev dataset: 15,451 refactoring instances across 12,256 PRs and 14,988 commits in real-world open-source Java projects, with agents (OpenAI Codex, Claude Code, Cursor named) targeting refactoring in 26.1% of commits. Finds agentic refactoring is dominated by low-level consistency edits (rename variable/parameter, change variable type) rather than the high-level design changes humans favor — a behavioral signature relevant to scope_creep (over-eager local edits) and incomplete_execution (rarely finishes a real architectural refactor). Relates to SAICA-KG failure modes: scope_creep, incomplete_execution. Primary citation for what agentic edits actually look like in production today.\n"
  },
  {
    "id": "jiang-2024-survey-llm-code",
    "authors": [
      "Juyong Jiang",
      "Fan Wang",
      "Jiasi Shen",
      "Sungju Kim",
      "Sunghun Kim"
    ],
    "year": 2024,
    "title": "A Survey on Large Language Models for Code Generation",
    "venue": "ACM Transactions on Software Engineering and Methodology (TOSEM)",
    "url": "https://arxiv.org/abs/2406.00515",
    "doi": "10.1145/3747588",
    "arxiv_id": "2406.00515",
    "semantic_scholar_id": "c8b18682965ff9dccc0130dab3d679f78cefa617",
    "abstract": "Large Language Models (LLMs) have garnered remarkable advancements across diverse code-related tasks, known as Code LLMs, particularly in code generation that generates source code with LLM from natural language descriptions. This burgeoning field has captured significant interest from both academic researchers and industry professionals due to its practical significance in software development, e.g., GitHub Copilot. Despite the active exploration of LLMs for a variety of code tasks, either from the perspective of Natural Language Processing (NLP) or Software Engineering (SE) or both, there is a noticeable absence of a comprehensive and up-to-date literature review dedicated to LLM for code generation. In this survey, we aim to bridge this gap by providing a systematic literature review that serves as a valuable reference for researchers investigating the cutting-edge progress in LLMs for code generation. We introduce a taxonomy to categorize and discuss the recent developments in LLMs for code generation, covering aspects such as data curation, latest advances, performance evaluation, ethical implications, environmental impact, and real-world applications. In addition, we present a historical overview of the evolution of LLMs for code generation and provide a quantitative and qualitative comparative analysis of experimental results of code LLMs, sourced from their original papers to ensure a fair comparison on the HumanEval, MBPP, and BigCodeBench benchmarks, across various levels of difficulty and types of programming tasks, to highlight the progressive enhancements in LLM capabilities for code generation. We identify critical challenges and promising opportunities regarding the gap between academia and practical development. Furthermore, we have established a dedicated resource GitHub page (https://github.com/juyongjiang/CodeLLMSurvey) to continuously document and disseminate the most recent advances in the field.",
    "tldr": "Systematic literature review of LLMs for code generation, organizing the field around data curation, advances, evaluation, ethics, environment, and real-world applications. Introduces a taxonomy of Code LLM developments and compares HumanEval/MBPP/BigCodeBench results across models.\n",
    "citation_count": 837,
    "relevance_tags": [
      "survey",
      "code-generation",
      "landscape",
      "background-citation"
    ],
    "notes": "Must-cite. Landscape anchor for SAICA-KG's background section — the canonical \"what is the state of LLMs for code\" reference when framing agentic-coding supervision as a downstream problem.\n"
  },
  {
    "id": "kota-2025-zero-trust-security-frameworks",
    "authors": [
      "S. Kota"
    ],
    "year": 2025,
    "title": "Zero-Trust Security Frameworks for Model Context Protocol (MCP) Server Communications",
    "venue": "International Journal of Innovative Research in Engineering &amp; Multidisciplinary Physical Sciences",
    "url": "https://www.semanticscholar.org/paper/90e083c73a326a4e553d9534c63470f99be413a5",
    "doi": "10.37082/ijirmps.v13.i6.232959",
    "semantic_scholar_id": "90e083c73a326a4e553d9534c63470f99be413a5",
    "abstract": "In the era of agentic artificial intelligence (AI), where large language models (LLMs) and autonomous agents invoke external tools and services through protocols such as the Model Context Protocol (MCP), the need for robust and rigorous security architectures is paramount. Conventional perimeter-based defenses are increasingly inadequate in these dynamic, distributed, and interconnected environments. This paper investigates the application of Zero-Trust Architecture (ZTA) principles to MCP server communications. We first analyze the threat surface introduced by MCP communications—the client-server interactions, tool invocations, context leakage, and lateral movement risks. We then propose a structured zero-trust framework tailored to the MCP ecosystem, encompassing identity and device verification, mutual authentication, micro-segmentation, dynamic policy enforcement, continuous monitoring, and breach containment. We provide a detailed architectural description, pseudo-code for policy evaluation, and complexity/scalability analysis. We also discuss deployment considerations in enterprise and cloud-native contexts. Our contribution is three-fold: (1) We bridge the gap between emerging AI agent-to-tool protocols (specifically MCP) and zero-trust security foundations. (2) We propose a formalised zero-trust framework for MCP communications. (3) We present an evaluative discussion of performance, scalability, strengths, and limitations. We conclude that zero-trust frameworks can significantly reduce risk in MCP deployments with manageable overhead, but real-world implementation requires rigorous identity, telemetry, and policy-management infrastructure.\n",
    "tldr": "This paper analyzes the threat surface introduced by MCP communications and proposes a structured zero-trust framework tailored to the MCP ecosystem, encompassing identity and device verification, mutual authentication, micro-segmentation, dynamic policy enforcement, continuous monitoring, and breach containment.\n",
    "relevance_tags": [
      "agentic-ai",
      "supervision",
      "MCP",
      "benchmark"
    ],
    "notes": "Position paper proposing a Zero-Trust Architecture (ZTA) framework for MCP server communications, covering identity and device verification, mutual authentication, micro-segmentation, dynamic policy enforcement, continuous monitoring, and breach containment. Names MCP-specific threat surfaces: client-server interactions, tool invocations, context leakage, and lateral movement. Relates to SAICA-KG failure modes: security_vulnerability, supply_chain_attack. Useful as a vocabulary anchor for the security-layer Tool nodes; light on empirical evidence so cite as theoretical framing rather than primary measurement.\n"
  },
  {
    "id": "krishna-2025-importing-phantoms-measuring-llm",
    "authors": [
      "Arjun Krishna",
      "Erick Galinkin",
      "Leon Derczynski",
      "Jeffrey Martin"
    ],
    "year": 2025,
    "title": "Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/473e0cc43e9c95be45e0d8a23fbe6d51cdfce485",
    "doi": "10.48550/arXiv.2501.19012",
    "arxiv_id": "2501.19012",
    "semantic_scholar_id": "473e0cc43e9c95be45e0d8a23fbe6d51cdfce485",
    "abstract": "Large Language Models (LLMs) have become an essential tool in the programmer's toolkit, but their tendency to hallucinate code can be used by malicious actors to introduce vulnerabilities to broad swathes of the software supply chain. In this work, we analyze package hallucination behaviour in LLMs across popular programming languages examining both existing package references and fictional dependencies. By analyzing this package hallucination behaviour we find potential attacks and suggest defensive strategies to defend against these attacks. We discover that package hallucination rate is predicated not only on model choice, but also programming language, model size, and specificity of the coding task request. The Pareto optimality boundary between code generation performance and package hallucination is sparsely populated, suggesting that coding models are not being optimized for secure code. Additionally, we find an inverse correlation between package hallucination rate and the HumanEval coding benchmark, offering a heuristic for evaluating the propensity of a model to hallucinate packages. Our metrics, findings and analyses provide a base for future models, securing AI-assisted software development workflows against package supply chain attacks.\n",
    "tldr": "This work analyzes package hallucination behaviour in LLMs across popular programming languages examining both existing package references and fictional dependencies, finding an inverse correlation between package hallucination rate and the HumanEval coding benchmark, offering a heuristic for evaluating the propensity of a model to hallucinate packages.\n",
    "citation_count": 7,
    "relevance_tags": [
      "benchmark",
      "supply-chain"
    ],
    "notes": "Measures package-hallucination behavior across multiple LLMs and programming languages. Two key empirical claims for SAICA-KG: (a) the Pareto frontier between code-generation performance and package-hallucination rate is sparsely populated, suggesting coding models are not currently optimized for secure code, and (b) there is an inverse correlation between package-hallucination rate and HumanEval performance, offering a heuristic for forecasting which models are most exposed. Relates to SAICA-KG failure modes: supply_chain_attack, fabrication, security_vulnerability. Cite as a complement to spracklen-2024 — same problem, different measurement angle.\n"
  },
  {
    "id": "krishnan-2025-advancing-multi-agent-systems",
    "authors": [
      "N. Krishnan"
    ],
    "year": 2025,
    "title": "Advancing Multi-Agent Systems Through Model Context Protocol: Architecture, Implementation, and Applications",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/66098cff05d84454084aa442640c98c076f375f2",
    "doi": "10.48550/arXiv.2504.21030",
    "arxiv_id": "2504.21030",
    "semantic_scholar_id": "66098cff05d84454084aa442640c98c076f375f2",
    "abstract": "Multi-agent systems represent a significant advancement in artificial intelligence, enabling complex problem-solving through coordinated specialized agents. However, these systems face fundamental challenges in context management, coordination efficiency, and scalable operation. This paper introduces a comprehensive framework for advancing multi-agent systems through Model Context Protocol (MCP), addressing these challenges through standardized context sharing and coordination mechanisms. We extend previous work on AI agent architectures by developing a unified theoretical foundation, advanced context management techniques, and scalable coordination patterns. Through detailed implementation case studies across enterprise knowledge management, collaborative research, and distributed problem-solving domains, we demonstrate significant performance improvements compared to traditional approaches. Our evaluation methodology provides a systematic assessment framework with benchmark tasks and datasets specifically designed for multi-agent systems. We identify current limitations, emerging research opportunities, and potential transformative applications across industries. This work contributes to the evolution of more capable, collaborative, and context-aware artificial intelligence systems that can effectively address complex real-world challenges.\n",
    "tldr": "A comprehensive framework for advancing multi-agent systems through Model Context Protocol (MCP), addressing these challenges through standardized context sharing and coordination mechanisms through standardized context sharing and coordination mechanisms is introduced.\n",
    "citation_count": 46,
    "relevance_tags": [
      "MCP",
      "benchmark"
    ],
    "notes": "Architectural framework paper proposing standardized context sharing and coordination mechanisms over the Model Context Protocol (MCP) for multi-agent systems. Mostly theoretical (case studies in enterprise knowledge management, collaborative research, and distributed problem-solving) but useful as an MCP-vocabulary anchor for SAICA-KG's tool-layer nodes. Relates to SAICA-KG failure modes: context_pollution, cascading_failure. Cite when discussing how MCP is intended to constrain context-sharing and where its design intent diverges from observed practice.\n"
  },
  {
    "id": "kumar-2026-agentforge",
    "authors": [
      "Rajesh Kumar",
      "Waqar Ali",
      "Junaid Ahmed",
      "Najmah Ali",
      "Shaban Usman"
    ],
    "year": 2026,
    "title": "AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2604.13120",
    "arxiv_id": "2604.13120",
    "semantic_scholar_id": "f019c8a7e42aecaa96ef3c4cbd2ecd3aaaa1ac0c",
    "abstract": "Large language models generate plausible code but cannot verify correctness. Existing multi-agent systems simulate execution or leave verification optional. We introduce execution-grounded verification as a first-class principle: every code change must survive sandboxed execution before propagation. We instantiate this principle in AGENTFORGE, a multi-agent framework where Planner, Coder, Tester, Debugger, and Critic agents coordinate through shared memory and a mandatory Docker sandbox. We formalize software engineering with LLMs as an iterative decision process over repository states, where execution feedback provides a stronger supervision signal than next-token likelihood. AGENTFORGE achieves 40.0\\% resolution on SWE-BENCH Lite, outperforming single-agent baselines by 26--28 points. Ablations confirm that execution feedback and role decomposition each independently drive performance. The framework is open-source at https://github.com/raja21068/AutoCodeAI.",
    "tldr": "Introduces execution-grounded verification as a first-class principle: every code change must survive sandboxed execution before propagation. AGENTFORGE instantiates this with Planner/Coder/Tester/Debugger/Critic agents sharing memory and a mandatory Docker sandbox, achieving 40.0% on SWE-Bench Lite — 26-28 points above single-agent baselines.\n",
    "citation_count": 0,
    "relevance_tags": [
      "multi-agent",
      "execution-grounded",
      "sandboxing",
      "coding-agent",
      "correction",
      "verification"
    ],
    "notes": "Must-cite. Canonical example of \"sandbox as first-class verification\" multi-agent architecture. Grounds SAICA-KG's correction ControlParadigm tools and the environment LocusOfControl; pairs with Yan 2025 (fault- tolerant sandboxing) for the sandbox Tool cluster.\n"
  },
  {
    "id": "lee-2025-hallucination-taxonomy",
    "authors": [
      "Yunseo Lee",
      "John Youngeun Song",
      "Dongsun Kim",
      "Jindae Kim",
      "Mijung Kim",
      "Jaechang Nam"
    ],
    "year": 2025,
    "title": "Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2504.20799",
    "doi": "10.48550/arXiv.2504.20799",
    "arxiv_id": "2504.20799",
    "semantic_scholar_id": "ab90297f5a95b79d67620c32d2474c9a1dfef8c8",
    "abstract": "Recent technical breakthroughs in large language models (LLMs) have enabled them to fluently generate source code. Software developers often leverage both general-purpose and code-specialized LLMs to revise existing code or even generate a whole function from scratch. These capabilities are also beneficial in no-code or low-code contexts, in which one can write programs without a technical background. However, due to their internal design, LLMs are prone to generating hallucinations, which are incorrect, nonsensical, and not justifiable information but difficult to identify its presence. This problem also occurs when generating source code. Once hallucinated code is produced, it is often challenging for users to identify and fix it, especially when such hallucinations can be identified under specific execution paths. As a result, the hallucinated code may remain unnoticed within the codebase. This survey investigates recent studies and techniques relevant to hallucinations generated by CodeLLMs. We categorize the types of hallucinations in the code generated by CodeLLMs, review existing benchmarks and mitigation strategies, and identify open challenges. Based on these findings, this survey outlines further research directions in the detection and removal of hallucinations produced by CodeLLMs.",
    "tldr": "Survey categorizing hallucination types in Code-LLM output, reviewing existing benchmarks and mitigation strategies, and identifying open challenges in detecting and removing hallucinated code.\n",
    "citation_count": 11,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "hallucination",
      "survey",
      "code-generation"
    ],
    "notes": "Must-cite. Secondary taxonomy anchor for the FailureMode axis — pairs with Tian 2024 (CodeHalu) and Zhang 2024 as the \"prior taxonomies\" set for the honest-MECE comparison.\n"
  },
  {
    "id": "li-2025-glue-code-protocols-critical",
    "authors": [
      "Qiaomu Li",
      "Ying Xie"
    ],
    "year": 2025,
    "title": "From Glue-Code to Protocols: A Critical Analysis of A2A and MCP Integration for Scalable Agent Systems",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/e790fbf3be688cc5c3cbda9c6fc6b15f1d3c2a49",
    "doi": "10.48550/arXiv.2505.03864",
    "arxiv_id": "2505.03864",
    "semantic_scholar_id": "e790fbf3be688cc5c3cbda9c6fc6b15f1d3c2a49",
    "abstract": "Artificial intelligence is rapidly evolving towards multi-agent systems where numerous AI agents collaborate and interact with external tools. Two key open standards, Google's Agent to Agent (A2A) protocol for inter-agent communication and Anthropic's Model Context Protocol (MCP) for standardized tool access, promise to overcome the limitations of fragmented, custom integration approaches. While their potential synergy is significant, this paper argues that effectively integrating A2A and MCP presents unique, emergent challenges at their intersection, particularly concerning semantic interoperability between agent tasks and tool capabilities, the compounded security risks arising from combined discovery and execution, and the practical governance required for the envisioned\"Agent Economy\". This work provides a critical analysis, moving beyond a survey to evaluate the practical implications and inherent difficulties of combining these horizontal and vertical integration standards. We examine the benefits (e.g., specialization, scalability) while critically assessing their dependencies and trade-offs in an integrated context. We identify key challenges increased by the integration, including novel security vulnerabilities, privacy complexities, debugging difficulties across protocols, and the need for robust semantic negotiation mechanisms. In summary, A2A+MCP offers a vital architectural foundation, but fully realizing its potential requires substantial advancements to manage the complexities of their combined operation.\n",
    "tldr": "This work provides a critical analysis, moving beyond a survey to evaluate the practical implications and inherent difficulties of combining these horizontal and vertical integration standards, and examines the benefits (e.g., specialization, scalability) while critically assessing their dependencies and trade-offs in an integrated context.\n",
    "citation_count": 13,
    "relevance_tags": [
      "MCP",
      "survey"
    ],
    "notes": "Critical analysis of integrating Google's Agent-to-Agent (A2A) protocol with Anthropic's Model Context Protocol (MCP). Names the emergent risk surfaces at the A2A+MCP intersection: semantic interoperability between agent tasks and tool capabilities, compounded security risks from combined discovery and execution, debugging difficulties across protocols, and governance gaps for the 'Agent Economy'. Relates to SAICA-KG failure modes: security_vulnerability, cascading_failure, scope_creep. Cite as supervision-vocabulary anchor when discussing inter-agent vs. agent-tool boundaries in the ControlParadigm axis.\n"
  },
  {
    "id": "li-2025-netmcp-network-aware-model",
    "authors": [
      "Enhan Li",
      "Hongyang Du",
      "Kaibin Huang"
    ],
    "year": 2025,
    "title": "NetMCP: Network-Aware Model Context Protocol Platform for LLM Capability Extension",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/5797ce51c448e9bc05869f1be77ce92b85e35761",
    "doi": "10.48550/arXiv.2510.13467",
    "arxiv_id": "2510.13467",
    "semantic_scholar_id": "5797ce51c448e9bc05869f1be77ce92b85e35761",
    "abstract": "Large Language Models (LLMs) remain static in functionality after training, and extending their capabilities requires integration with external data, computation, and services. The Model Context Protocol (MCP) has emerged as a standard interface for such extensions, but current implementations rely solely on semantic matching between users'requests and server function descriptions, which makes current deployments and simulation testbeds fragile under latency fluctuations or server failures. We address this gap by enhancing MCP tool routing algorithms with real-time awareness of network and server status. To provide a controlled test environment for development and evaluation, we construct a heterogeneous experimental platform, namely Network-aware MCP (NetMCP), which offers five representative network states and build a benchmark for latency sequence generation and MCP server datasets. On top of NetMCP platform, we analyze latency sequences and propose a Semantic-Oriented and Network-Aware Routing (SONAR) algorithm, which jointly optimizes semantic similarity and network Quality of Service (QoS) metrics for adaptive tool routing. Results show that SONAR consistently improves task success rate and reduces completion time and failure number compared with semantic-only, LLM-based baselines, demonstrating the value of network-aware design for production-scale LLM systems. The code for NetMCP is available at https://github.com/NICE-HKU/NetMCP.\n",
    "tldr": "Results show that SONAR consistently improves task success rate and reduces completion time and failure number compared with semantic-only, LLM-based baselines, demonstrating the value of network-aware design for production-scale LLM systems.\n",
    "citation_count": 4,
    "relevance_tags": [
      "MCP",
      "benchmark"
    ],
    "notes": "Builds NetMCP, an experimental MCP testbed over five representative network states, and proposes SONAR (Semantic Oriented and Network Aware Routing), an MCP tool-routing algorithm that jointly optimizes semantic similarity and network QoS. Demonstrates that semantic-only routing leaves task success and latency on the table when MCP servers degrade or fail. Relates to SAICA-KG failure modes: incomplete_execution, cascading_failure. Useful as a production-engineering anchor: even when the agent reasons correctly, infrastructure-layer faults propagate as task-level failures, which informs the supervision boundary between tool-call layer and agent layer. Code at github.com/NICE-HKU/NetMCP.\n"
  },
  {
    "id": "li-2026-security-considerations-artificial-intelligence",
    "authors": [
      "Ninghui Li",
      "Kaiyuan Zhang",
      "Kyle Polley",
      "Jerry Ma"
    ],
    "year": 2026,
    "title": "Security Considerations for Artificial Intelligence Agents",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/91abd24806296eb10748aec15af67f4ee0b43ee1",
    "arxiv_id": "2603.12230",
    "semantic_scholar_id": "91abd24806296eb10748aec15af67f4ee0b43ee1",
    "abstract": "This article, a lightly adapted version of Perplexity's response to NIST/CAISI Request for Information 2025-0035, details our observations and recommendations concerning the security of frontier AI agents. These insights are informed by Perplexity's experience operating general-purpose agentic systems used by millions of users and thousands of enterprises in both controlled and open-world environments. Agent architectures change core assumptions around code-data separation, authority boundaries, and execution predictability, creating new confidentiality, integrity, and availability failure modes. We map principal attack surfaces across tools, connectors, hosting boundaries, and multi-agent coordination, with particular emphasis on indirect prompt injection, confused-deputy behavior, and cascading failures in long-running workflows. We then assess current defenses as a layered stack: input-level and model-level mitigations, sandboxed execution, and deterministic policy enforcement for high-consequence actions. Finally, we identify standards and research gaps, including adaptive security benchmarks, policy models for delegation and privilege control, and guidance for secure multi-agent system design aligned with NIST risk management principles.\n",
    "tldr": "This article details observations and recommendations concerning the security of frontier AI agents, and identifies standards and research gaps, including adaptive security benchmarks, policy models for delegation and privilege control, and guidance for secure multi-agent system design aligned with NIST risk management principles.\n",
    "relevance_tags": [
      "failure-mode-taxonomy",
      "agentic-ai",
      "benchmark",
      "guardrails"
    ],
    "notes": "Lightly-adapted version of Perplexity's response to NIST/CAISI RFI 2025-0035 on frontier AI agent security. Maps principal attack surfaces across tools, connectors, hosting boundaries, and multi-agent coordination, with explicit emphasis on indirect prompt injection, confused-deputy behavior, and cascading failures in long-running workflows. Assesses defenses as a layered stack: input/model-level mitigations, sandboxed execution, and deterministic policy enforcement for high-consequence actions. Relates to SAICA-KG failure modes: security_vulnerability, cascading_failure, supply_chain_attack. Useful as an industry-perspective anchor for the layered-defense framing in SAICA-KG's ControlParadigm axis.\n"
  },
  {
    "id": "lindner-2025-monitoring",
    "authors": [
      "David Lindner",
      "Charlie Griffin",
      "Tomasz Korbak",
      "Roland S. Zimmermann",
      "Geoffrey Irving",
      "Sebastian Farquhar",
      "Alan Cooney"
    ],
    "year": 2025,
    "title": "Practical challenges of control monitoring in frontier AI deployments",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2512.22154",
    "doi": "10.48550/arXiv.2512.22154",
    "arxiv_id": "2512.22154",
    "semantic_scholar_id": "03885f500d3c8bb54443ac94df8a46110e5b68b9",
    "abstract": "Automated control monitors could play an important role in overseeing highly capable AI agents that we do not fully trust. Prior work has explored control monitoring in simplified settings, but scaling monitoring to real-world deployments introduces additional dynamics: parallel agent instances, non-negligible oversight latency, incremental attacks between agent instances, and the difficulty of identifying scheming agents based on individual harmful actions. In this paper, we analyse design choices to address these challenges, focusing on three forms of monitoring with different latency-safety trade-offs: synchronous, semi-synchronous, and asynchronous monitoring. We introduce a high-level safety case sketch as a tool for understanding and comparing these monitoring protocols. Our analysis identifies three challenges -- oversight, latency, and recovery -- and explores them in four case studies of possible future AI deployments.",
    "tldr": "Analyzes three forms of automated control monitoring with different latency-safety trade-offs: synchronous, semi-synchronous, and asynchronous. Introduces a high-level safety-case sketch and identifies three core challenges (oversight, latency, recovery) across four future-deployment case studies.\n",
    "citation_count": 2,
    "relevance_tags": [
      "monitoring",
      "temporal-phase",
      "control-paradigm",
      "supervision-vocabulary",
      "safety-case"
    ],
    "notes": "Must-cite. The sync / semi-sync / async triple is the canonical external alignment for SAICA-KG's TemporalPhase partition (pre_generation / in_generation / post_generation) — see enums.yml. Cite for external validation of the TemporalPhase axis and for the oversight/latency/ recovery vocabulary used across Tool nodes.\n"
  },
  {
    "id": "liu-2024-beyond-functional-correctness",
    "authors": [
      "Fang Liu",
      "Yang Liu",
      "Lin Shi",
      "Houkun Yang",
      "Zhen Zhang"
    ],
    "year": 2024,
    "title": "Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code",
    "venue": "IEEE Transactions on Software Engineering",
    "url": "https://www.semanticscholar.org/paper/31a23ae2821910f6b26d8991e953d20ae6d2297f",
    "abstract": "The rise of Large Language Models (LLMs) has significantly advanced various applications on software engineering tasks, particularly in code generation. Despite the promising performance, LLMs are prone to generate hallucinations, which means LLMs might produce outputs that deviate from users’ intent, exhibit internal inconsistencies, or misaligned with the real-world knowledge, making the deployment of LLMs potentially risky in a wide range of applications. Existing work mainly focuses on investigating the hallucination in the domain of Natural Language Generation (NLG), leaving a gap in comprehensively understanding the types, causes, and impacts of hallucinations in the context of code generation. To bridge the gap, we conducted a thematic analysis of the LLM-generated code to summarize and categorize the hallucinations, as well as their causes and impacts. Our study established a comprehensive taxonomy of code hallucinations, encompassing 3 primary categories and 12 specific categories. Furthermore, we systematically analyzed the distribution of hallucinations, exploring variations among different LLMs and benchmarks. Moreover, we perform an in-depth analysis on the causes and impacts of various hallucinations, aiming to provide valuable insights into hallucination mitigation. Finally, to enhance the correctness and reliability of LLM-generated code in a lightweight manner, we explore training-free hallucination mitigation approaches by prompt enhancing techniques. We believe our findings will shed light on future research about code hallucination evaluation and mitigation, ultimately paving the way for building more effective and reliable code LLMs in the future. The replication package is available at https://github.com/Lorien1128/code_hallucination.",
    "tldr": "Empirical study and taxonomy of hallucinations in LLM-generated code, with 3/12 category classification that is the closest prior art to SAICA-KG's FailureMode axis. Conflates axes (cause, symptom, type) rather than partitioning them orthogonally.\n",
    "citation_count": 216,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "hallucination",
      "code-generation"
    ],
    "notes": "Must-cite. Anchor paper for FailureMode taxonomy discussion and for the \"prior taxonomies are not MECE\" argument in the SAICA-KG paper.\n"
  },
  {
    "id": "liu-2025-code-copycat-conundrum-demystifying",
    "authors": [
      "Mingwei Liu",
      "Juntao Li",
      "Ying Wang",
      "Xueying Du",
      "Zuoyu Ou",
      "Qiuyuan Chen",
      "Bingxu An",
      "Zhao Wei",
      "Yong Xu",
      "Fang Zou",
      "Xing Peng",
      "Yiling Lou"
    ],
    "year": 2025,
    "title": "Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generation",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/9abb51ba135ccec7902d06777efc2f60ecfe262f",
    "doi": "10.48550/arXiv.2504.12608",
    "arxiv_id": "2504.12608",
    "semantic_scholar_id": "9abb51ba135ccec7902d06777efc2f60ecfe262f",
    "abstract": "Despite recent advances in Large Language Models (LLMs) for code generation, the quality of LLM-generated code still faces significant challenges. One significant issue is code repetition, which refers to the model's tendency to generate structurally redundant code, resulting in inefficiencies and reduced readability. To address this, we conduct the first empirical study to investigate the prevalence and nature of repetition across 19 state-of-the-art code LLMs using three widely-used benchmarks. Our study includes both quantitative and qualitative analyses, revealing that repetition is pervasive and manifests at various granularities and extents, including character, statement, and block levels. We further summarize a taxonomy of 20 repetition patterns. Building on our findings, we propose DeRep, a rule-based technique designed to detect and mitigate repetition in generated code. We evaluate DeRep using both open-source benchmarks and in an industrial setting. Our results demonstrate that DeRep significantly outperforms baselines in reducing repetition (with an average improvements of 91.3%, 93.5%, and 79.9% in rep-3, rep-line, and sim-line metrics) and enhancing code quality (with a Pass@1 increase of 208.3% over greedy search). Furthermore, integrating DeRep improves the performance of existing repetition mitigation methods, with Pass@1 improvements ranging from 53.7% to 215.7%.\n",
    "tldr": "The first empirical study to investigate the prevalence and nature of repetition across 19 state-of-the-art code LLMs using three widely-used benchmarks finds that repetition is pervasive and manifests at various granularities and extents, including character, statement, and block levels.\n",
    "citation_count": 5,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "benchmark",
      "empirical"
    ],
    "notes": "First empirical study of code repetition across 19 SOTA code LLMs on three benchmarks; produces a 20-pattern taxonomy of repetition at character, statement, and block granularity. Proposes DeRep, a rule-based detect-and-mitigate technique that delivers large repetition reductions and a 208.3% Pass@1 uplift over greedy search. Repetition is a concrete signature of context_pollution (the model echoing its own context) and of degraded logic_error patterns (e.g. duplicated branches hiding bugs). Relates to SAICA-KG failure modes: context_pollution, logic_error. Cite when arguing that syntactic-quality detectors complement semantic ones in the supervision stack.\n"
  },
  {
    "id": "liu-2025-code-copycat",
    "authors": [
      "Mingwei Liu",
      "Yanjun Li",
      "Chong Wang",
      "Xin Du",
      "Tao Ou"
    ],
    "year": 2025,
    "title": "The Code Copycat Conundrum: Repetition in LLM Code Generation",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/9abb51ba135ccec7902d06777efc2f60ecfe262f",
    "tldr": "Empirical study of intra-generation repetition in LLM code output. The closest semantic neighbor to SAICA-KG's dependency_blindness class, but studies repetition *within* a single generation rather than cross-codebase reimplementation of existing library code.\n",
    "citation_count": 5,
    "relevance_tags": [
      "repetition",
      "reuse",
      "dependency-blindness-adjacent"
    ],
    "notes": "Cite as the contrast paper for dependency_blindness. SAICA-KG should explicitly distinguish: Liu 2025 measures repetition *within* generated output; dependency_blindness is about the agent re-implementing code that already exists elsewhere in the declared dependency graph.\n"
  },
  {
    "id": "liu-2025-empirical-study-failures-automated",
    "authors": [
      "Simiao Liu",
      "Fang Liu",
      "Liehao Li",
      "Xin Tan",
      "Yinghao Zhu",
      "Xiaoli Lian",
      "Li Zhang"
    ],
    "year": 2025,
    "title": "An Empirical Study on Failures in Automated Issue Solving",
    "venue": "arXiv",
    "url": "https://arxiv.org/pdf/2509.13941",
    "doi": "10.48550/arXiv.2509.13941",
    "arxiv_id": "2509.13941",
    "semantic_scholar_id": "281333210",
    "abstract": "Automated issue solving seeks to autonomously identify and repair defective code snippets across an entire codebase. SWE-Bench has emerged as the most widely adopted benchmark for evaluating progress in this area. While LLM-based agentic tools show great promise, they still fail on a substantial portion of tasks. Moreover, current evaluations primarily report aggregate issue-solving rates, which obscure the underlying causes of success and failure, making it challenging to diagnose model weaknesses or guide targeted improvements. To bridge this gap, we first analyze the performance and efficiency of three SOTA tools, spanning both pipeline-based and agentic architectures, in automated issue solving tasks of SWE-Bench-Verified under varying task characteristics. Furthermore, to move from high-level performance metrics to underlying cause analysis, we conducted a systematic manual analysis of 150 failed instances. From this analysis, we developed a comprehensive taxonomy of failure modes comprising 3 primary phases, 9 main categories, and 25 fine-grained subcategories. Then we systematically analyze the distribution of the identified failure modes, the results reveal distinct failure fingerprints between the two architectural paradigms, with the majority of agentic failures stemming from flawed reasoning and cognitive deadlocks. Motivated by these insights, we propose a collaborative Expert-Executor framework. It introduces a supervisory Expert agent tasked with providing strategic oversight and course-correction for a primary Executor agent. This architecture is designed to correct flawed reasoning and break the cognitive deadlocks that frequently lead to failure. Experiments show that our framework solves 22.2% of previously intractable issues for a leading single agent. These findings pave the way for building more robust agents through diagnostic evaluation and collaborative design.\n",
    "tldr": "Automated issue solving seeks to autonomously identify and repair defective code snippets across an entire codebase. SWE-Bench has emerged as the most widely adopted benchmark for evaluating progress in this area.\n",
    "citation_count": 5,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "coding-agent",
      "agentic-ai",
      "supervision",
      "benchmark",
      "empirical"
    ],
    "notes": "Manual analysis of 150 failed SWE-Bench-Verified instances across three SOTA tools (pipeline-based and agentic) produces a taxonomy of 3 phases, 9 main categories, and 25 fine-grained failure subcategories. Key finding: agentic failures are dominated by flawed reasoning and 'cognitive deadlocks' — distinct failure fingerprints from pipeline-based tools. Proposes a collaborative Expert-Executor framework where a supervisory Expert agent provides strategic oversight and course-correction, solving 22.2% of previously intractable issues. Strong primary citation for SAICA-KG's logic_error and incomplete_execution modes; the cognitive-deadlock finding is direct evidence for incomplete_execution. Relates to SAICA-KG failure modes: logic_error, incomplete_execution, cascading_failure.\n"
  },
  {
    "id": "liu-2026-packmonitor-enabling-zero-package",
    "authors": [
      "Xiting Liu",
      "Yuetong Liu",
      "Yitong Zhang",
      "Jia Li",
      "Shiyuan Hu"
    ],
    "year": 2026,
    "title": "PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/31191f5478cac72e8dc816fbc5c46ce00f98ddf4",
    "doi": "10.48550/arXiv.2602.20717",
    "arxiv_id": "2602.20717",
    "semantic_scholar_id": "31191f5478cac72e8dc816fbc5c46ce00f98ddf4",
    "abstract": "As Large Language Models (LLMs) are increasingly integrated into software development workflows, their trustworthiness has become a critical concern. However, in dependency recommendation scenarios, the reliability of LLMs is undermined by widespread package hallucinations, where models often recommend hallucinated packages. Recent studies have proposed a range of approaches to mitigate this issue. Nevertheless, existing approaches typically merely reduce hallucination rates rather than eliminate them, leaving persistent software security risks. In this work, we argue that package hallucinations are theoretically preventable based on the key insight that package validity is decidable through finite and enumerable authoritative package lists. Building on this, we propose PackMonitor, the first approach capable of fundamentally eliminating package hallucinations by continuously monitoring the model's decoding process and intervening when necessary. To implement this in practice, PackMonitor addresses three key challenges: (1) determining when to trigger intervention via a Context-Aware Parser that continuously monitors model outputs and selectively activates intervening only during installation command generation; (2) resolving how to intervene by employing a Package-Name Intervenor that strictly limits the decoding space to an authoritative package list; and (3) ensuring monitoring efficiency through a DFA-Caching Mechanism that enables scalability to millions of packages with negligible overhead. Extensive experiments on five widely used LLMs demonstrate that PackMonitor is a training-free, plug-and-play solution that consistently reduces package hallucination rates to zero while maintaining low-latency inference and preserving original model capabilities.\n",
    "tldr": "Extensive experiments on five widely used LLMs demonstrate that PackMonitor is a training-free, plug-and-play solution that consistently reduces package hallucination rates to zero while maintaining low-latency inference and preserving original model capabilities.\n",
    "relevance_tags": [
      "supervision",
      "supply-chain"
    ],
    "notes": "Introduces PackMonitor, a decoding-time monitor that eliminates (not just mitigates) package hallucinations by constraining the decoding space to an authoritative package list when the agent is generating an install command. Three mechanisms: a Context-Aware Parser that triggers intervention only during install-command generation, a Package-Name Intervenor that restricts decoding, and a DFA-Caching mechanism that scales to millions of packages. Tested on five widely used LLMs, drives package-hallucination rate to zero with negligible latency overhead. Direct demonstration that in_generation TemporalPhase supervision can fully prevent a specific FailureMode. Relates to SAICA-KG failure modes: supply_chain_attack, fabrication, dependency_blindness. Strong primary cite for the in_generation phase of the TemporalPhase axis.\n"
  },
  {
    "id": "luo-2025-mcp-universe",
    "authors": [
      "Ziyang Luo",
      "Zhiqi Shen",
      "Wenzhuo Yang",
      "Zirui Zhao",
      "Prathyusha Jwalapuram",
      "Amrita Saha",
      "Doyen Sahoo",
      "Silvio Savarese",
      "Caiming Xiong",
      "Junnan Li"
    ],
    "year": 2025,
    "title": "MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2508.14704",
    "doi": "10.48550/arXiv.2508.14704",
    "arxiv_id": "2508.14704",
    "semantic_scholar_id": "6935b713ca7e42dfa2ec93b0adacc16b291edc1d",
    "abstract": "The Model Context Protocol has emerged as a transformative standard for connecting large language models to external data sources and tools, rapidly gaining adoption across major AI providers and development platforms. However, existing benchmarks are overly simplistic and fail to capture real application challenges such as long-horizon reasoning and large, unfamiliar tool spaces. To address this critical gap, we introduce MCP-Universe, the first comprehensive benchmark specifically designed to evaluate LLMs in realistic and hard tasks through interaction with real-world MCP servers. Our benchmark encompasses 6 core domains spanning 11 different MCP servers: Location Navigation, Repository Management, Financial Analysis, 3D Design, Browser Automation, and Web Searching. To ensure rigorous evaluation, we implement execution-based evaluators, including format evaluators for agent format compliance, static evaluators for time-invariant content matching, and dynamic evaluators that automatically retrieve real-time ground truth for temporally sensitive tasks. Through extensive evaluation of leading LLMs, we find that even SOTA models such as GPT-5 (43.72%), Grok-4 (33.33%) and Claude-4.0-Sonnet (29.44%) exhibit significant performance limitations. In addition, our benchmark poses a significant long-context challenge for LLM agents, as the number of input tokens increases rapidly with the number of interaction steps. Moreover, it introduces an unknown-tools challenge, as LLM agents often lack familiarity with the precise usage of the MCP servers. Notably, enterprise-level agents like Cursor cannot achieve better performance than standard ReAct frameworks. Beyond evaluation, we open-source our extensible evaluation framework with UI support, enabling researchers and practitioners to seamlessly integrate new agents and MCP servers while fostering innovation in the rapidly evolving MCP ecosystem.",
    "tldr": "First comprehensive benchmark of LLM agents against real-world MCP servers across 6 domains and 11 MCP servers. Uses execution-based evaluators (format, static, dynamic). Even GPT-5 reaches only 43.72%; exposes the long-context and unknown-tools challenges inherent to MCP.\n",
    "citation_count": 63,
    "relevance_tags": [
      "benchmark",
      "MCP",
      "agent-tool",
      "real-world-evaluation",
      "supply-chain-adjacent"
    ],
    "notes": "Must-cite. Canonical MCP benchmark as of 2025. Grounds SAICA-KG's supply_chain_attack and dependency_blindness modes with measured failure rates against real MCP servers; anchor for any MCP-keyed Tool nodes.\n"
  },
  {
    "id": "manheim-homewood-2025-control-oversight",
    "authors": [
      "David Manheim",
      "James Homewood"
    ],
    "year": 2025,
    "title": "Distinguishing Control and Oversight in Agentic AI",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2507.03525",
    "arxiv_id": "2507.03525",
    "abstract": "Oversight and control, which we collectively call supervision, are often discussed as ways to ensure that AI systems are accountable, reliable, and able to fulfill governance and management requirements. However, the requirements for \"human oversight\" risk codifying vague or inconsistent interpretations of key concepts like oversight and control. This ambiguous terminology could undermine efforts to design or evaluate systems that must operate under meaningful human supervision. This matters because the term is used by regulatory texts such as the EU AI Act. This paper undertakes a targeted critical review of literature on supervision outside of AI, along with a brief summary of past work on the topic related to AI. We next differentiate control as ex-ante or real-time and operational rather than policy or governance, and oversight as performed ex-post, or a policy and governance function. Control aims to prevent failures, while oversight focuses on detection, remediation, or incentives for future prevention. Building on this, we make three contributions. 1) We propose a framework to align regulatory expectations with what is technically and organizationally plausible, articulating the conditions under which each mechanism is possible, where they fall short, and what is required to make them meaningful in practice. 2) We outline how supervision methods should be documented and integrated into risk management, and drawing on the Microsoft Responsible AI Maturity Model, we outline a maturity model for AI supervision. 3) We explicitly highlight boundaries of these mechanisms, including where they apply, where they fail, and where it is clear that no existing methods suffice. This foregrounds the question of whether meaningful supervision is possible in a given deployment context, and can support regulators, auditors, and practitioners in identifying both present and future limitations.",
    "tldr": "Distinguishes control (ex-ante / operational) from oversight (ex-post / governance). Provides vocabulary that converges with SAICA-KG's TemporalPhase and ControlParadigm partitions.\n",
    "relevance_tags": [
      "supervision-vocabulary",
      "control-paradigm",
      "temporal-phase"
    ],
    "notes": "Cite for external validation of the TemporalPhase partition and for the ControlParadigm discussion. Useful anchor against \"your partitions are arbitrary\" critiques.\n"
  },
  {
    "id": "morabito-wu-2025-code-autopilot",
    "authors": [
      "Roberto Morabito",
      "Guanghan Wu"
    ],
    "year": 2025,
    "title": "When the Code Autopilot Breaks: Why LLMs Falter in Embedded Machine Learning",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2509.10946",
    "doi": "10.48550/arXiv.2509.10946",
    "arxiv_id": "2509.10946",
    "semantic_scholar_id": "6a8e86bed0464f16e7604a43c7bd277e89467f23",
    "abstract": "Large Language Models (LLMs) are increasingly used to automate software generation in embedded machine learning workflows, yet their outputs often fail silently or behave unpredictably. This article presents an empirical investigation of failure modes in LLM-powered ML pipelines, based on an autopilot framework that orchestrates data preprocessing, model conversion, and on-device inference code generation. We show how prompt format, model behavior, and structural assumptions influence both success rates and failure characteristics, often in ways that standard validation pipelines fail to detect. Our analysis reveals a diverse set of error-prone behaviors, including format-induced misinterpretations and runtime-disruptive code that compiles but breaks downstream. We derive a taxonomy of failure categories and analyze errors across multiple LLMs, highlighting common root causes and systemic fragilities.\n",
    "tldr": "Empirical study of LLM code-generation failure modes in embedded ML pipelines; derives a failure-category taxonomy emphasizing format-induced misinterpretations and runtime-disruptive-but-compiling code.\n",
    "citation_count": 0,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "coding-agent",
      "embedded-ml",
      "empirical"
    ],
    "notes": "Domain-specific failure taxonomy (embedded ML) that exposes compiles-but-breaks-downstream patterns relevant to SAICA-KG's logic_error and obsolescence modes. Narrower than the general-agent taxonomies but useful for triangulating silent-failure signals.\n"
  },
  {
    "id": "navneet-2025-safe-ai",
    "authors": [
      "Satyam Kumar Navneet",
      "Joydeep Chandra"
    ],
    "year": 2025,
    "title": "Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2508.11824",
    "doi": "10.48550/arXiv.2508.11824",
    "arxiv_id": "2508.11824",
    "semantic_scholar_id": "79b937dcee3b2583e66df5c99d9f64734bf313c1",
    "abstract": "The integration of Large Language Models (LLMs) into software engineering has revolutionized code generation, enabling unprecedented productivity through promptware and autonomous AI agents. However, this transformation introduces significant risks, including insecure code generation, hallucinated outputs, irreversible actions, and a lack of transparency and accountability. Incidents like the Replit database deletion underscore the urgent need for robust safety and governance mechanisms. This paper comprehensively analyzes the inherent challenges of LLM-assisted code generation, such as vulnerability inheritance, overtrust, misinterpretation, and the absence of standardized validation and rollback protocols. To address these, we propose the SAFE-AI Framework, a holistic approach emphasizing Safety, Auditability, Feedback, and Explainability. The framework integrates guardrails, sandboxing, runtime verification, risk-aware logging, human-in-the-loop systems, and explainable AI techniques to mitigate risks while fostering trust and compliance. We introduce a novel taxonomy of AI behaviors categorizing suggestive, generative, autonomous, and destructive actions to guide risk assessment and oversight. Additionally, we identify open problems, including the lack of standardized benchmarks for code specific hallucinations and autonomy levels, and propose future research directions for hybrid verification, semantic guardrails, and proactive governance tools. Through detailed comparisons of autonomy control, prompt engineering, explainability, and governance frameworks, this paper provides a roadmap for responsible AI integration in software engineering, aligning with emerging regulations like the EU AI Act and Canada's AIDA to ensure safe, transparent, and accountable AI-driven development.",
    "tldr": "Analyzes failure classes in LLM-assisted code generation (vulnerability inheritance, overtrust, misinterpretation, absence of validation/rollback) and proposes the SAFE-AI framework (Safety, Auditability, Feedback, Explainability). Introduces a suggestive / generative / autonomous / destructive taxonomy of AI behaviors to guide oversight.\n",
    "citation_count": 5,
    "relevance_tags": [
      "autonomy",
      "supervision-vocabulary",
      "failure-mode-taxonomy",
      "governance",
      "destructive-action"
    ],
    "notes": "Must-cite. The suggestive / generative / autonomous / destructive axis is the canonical external alignment for SAICA-KG's AutonomyLevel enum — see enums.yml. Cite for external validation of the AutonomyLevel partition and for the destructive-action failure framing that grounds recovery ControlParadigm Tools.\n"
  },
  {
    "id": "ni-2025-viscoder2-building-multi-language",
    "authors": [
      "Yuansheng Ni",
      "Songcheng Cai",
      "Xiang Chen",
      "Jiarong Liang",
      "Z. Lyu",
      "Jiaqi Deng",
      "Kai Zou",
      "Ping Nie",
      "Fei Yuan",
      "Xiang Yue",
      "Wenhu Chen"
    ],
    "year": 2025,
    "title": "VisCoder2: Building Multi-Language Visualization Coding Agents",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/86366ae838f8b9e3e4f7dfa0ebb218f17b305142",
    "doi": "10.48550/arXiv.2510.23642",
    "arxiv_id": "2510.23642",
    "semantic_scholar_id": "86366ae838f8b9e3e4f7dfa0ebb218f17b305142",
    "abstract": "Large language models (LLMs) have recently enabled coding agents capable of generating, executing, and revising visualization code. However, existing models often fail in practical workflows due to limited language coverage, unreliable execution, and lack of iterative correction mechanisms. Progress has been constrained by narrow datasets and benchmarks that emphasize single-round generation and single-language tasks. To address these challenges, we introduce three complementary resources for advancing visualization coding agents. VisCode-Multi-679K is a large-scale, supervised dataset containing 679K validated and executable visualization samples with multi-turn correction dialogues across 12 programming languages. VisPlotBench is a benchmark for systematic evaluation, featuring executable tasks, rendered outputs, and protocols for both initial generation and multi-round self-debug. Finally, we present VisCoder2, a family of multi-language visualization models trained on VisCode-Multi-679K. Experiments show that VisCoder2 significantly outperforms strong open-source baselines and approaches the performance of proprietary models like GPT-4.1, with further gains from iterative self-debug, reaching 82.4% overall execution pass rate at the 32B scale, particularly in symbolic or compiler-dependent languages.\n",
    "tldr": "Experiments show that VisCoder2 significantly outperforms strong open-source baselines and approaches the performance of proprietary models like GPT-4.1, with further gains from iterative self-debug, reaching 82.4% overall execution pass rate at the 32B scale, particularly in symbolic or compiler-dependent languages.\n",
    "citation_count": 2,
    "relevance_tags": [
      "coding-agent",
      "supervision",
      "benchmark"
    ],
    "notes": "Releases three resources for visualization-coding agents: VisCode-Multi-679K (679K validated executable visualization samples with multi-turn correction dialogues across 12 languages), VisPlotBench (an executable benchmark with single-shot and multi-round self-debug protocols), and VisCoder2 (a model family trained on the dataset). Reports 82.4% overall execution pass rate at 32B with iterative self-debug, particularly in symbolic and compiler-dependent languages. Useful as a worked example of how iterative self-debug closes the gap on logic_error and incomplete_execution failures, and as a Tool-node citation for the visualization-coding subdomain. Relates to SAICA-KG failure modes: logic_error, incomplete_execution.\n"
  },
  {
    "id": "piao-2025-agentbay-hybrid-interaction-sandbox",
    "authors": [
      "Y. Piao",
      "Hongbo Min",
      "Hang Su",
      "Leilei Zhang",
      "Lei Wang",
      "Yue Yin",
      "Xiao Wu",
      "Zhe Xu",
      "Liwei Qu",
      "Hang Li",
      "Xinxin Zeng",
      "Wei Tian",
      "Fei Yu",
      "Xiaowei Li",
      "Jiayi Jiang",
      "Tongxuan Liu",
      "Haoxiang Tian",
      "Yufei Que",
      "Xiao-Qing Tu",
      "Bing Suo",
      "Yuebing Li",
      "Xiang-Hui Chen",
      "Ze-Yu Zhao",
      "Jia-Xiang Tang",
      "Wei Huang",
      "Xuguang Li",
      "Jing Zhao",
      "Jin Li",
      "Jie Shen",
      "Jinkui Ren",
      "Xiantao Zhang"
    ],
    "year": 2025,
    "title": "AgentBay: A Hybrid Interaction Sandbox for Seamless Human-AI Intervention in Agentic Systems",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/27a3bd1914156114cf093099241ffa86819190f1",
    "doi": "10.48550/arXiv.2512.04367",
    "arxiv_id": "2512.04367",
    "semantic_scholar_id": "27a3bd1914156114cf093099241ffa86819190f1",
    "abstract": "The rapid advancement of Large Language Models (LLMs) is catalyzing a shift towards autonomous AI Agents capable of executing complex, multi-step tasks. However, these agents remain brittle when faced with real-world exceptions, making Human-in-the-Loop (HITL) supervision essential for mission-critical applications. In this paper, we present AgentBay, a novel sandbox service designed from the ground up for hybrid interaction. AgentBay provides secure, isolated execution environments spanning Windows, Linux, Android, Web Browsers, and Code interpreters. Its core contribution is a unified session accessible via a hybrid control interface: An AI agent can interact programmatically via mainstream interfaces (MCP, Open Source SDK), while a human operator can, at any moment, seamlessly take over full manual control. This seamless intervention is enabled by Adaptive Streaming Protocol (ASP). Unlike traditional VNC/RDP, ASP is specifically engineered for this hybrid use case, delivering an ultra-low-latency, smoother user experience that remains resilient even in weak network environments. It achieves this by dynamically blending command-based and video-based streaming, adapting its encoding strategy based on network conditions and the current controller (AI or human). Our evaluation demonstrates strong results in security, performance, and task completion rates. In a benchmark of complex tasks, the AgentBay (Agent + Human) model achieved more than 48% success rate improvement. Furthermore, our ASP protocol reduces bandwidth consumption by up to 50% compared to standard RDP, and in end-to-end latency with around 5% reduction, especially under poor network conditions. We posit that AgentBay provides a foundational primitive for building the next generation of reliable, human-supervised autonomous systems.\n",
    "tldr": "It is proposed that AgentBay provides a foundational primitive for building the next generation of reliable, human-supervised autonomous systems, especially under poor network conditions.\n",
    "citation_count": 1,
    "relevance_tags": [
      "agentic-ai",
      "supervision",
      "MCP",
      "benchmark",
      "guardrails"
    ],
    "notes": "Introduces AgentBay, a sandbox service providing isolated execution environments across Windows, Linux, Android, web browsers, and code interpreters with a hybrid AI-or-human control interface (MCP / SDK programmatic access for agents; seamless human takeover via an Adaptive Streaming Protocol). Reports >48% success-rate improvement on complex tasks for the Agent+Human model. Direct evidence that HITL-with-low-friction-takeover is a measurable supervision lever. Relates to SAICA-KG failure modes: incomplete_execution, scope_creep. Useful as a Tool-node citation for the sandbox category and as evidence that the human-takeover latency budget is itself a supervision design parameter (echoing Lindner-2025's sync/semi-sync/async axis).\n"
  },
  {
    "id": "qi-2026-rift",
    "authors": [
      "Zhengyang Qi",
      "Charles Dickens",
      "Derek Pham",
      "A. Dsouza",
      "Armin Parchami",
      "Frederic Sala",
      "P. Varma"
    ],
    "year": 2026,
    "title": "RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2604.01375",
    "arxiv_id": "2604.01375",
    "semantic_scholar_id": "5e99dc492a86cc1df499e9b8ca62cc9e3af16c1e",
    "abstract": "Rubric-based evaluation is widely used in LLM benchmarks and training pipelines for open-ended, less verifiable tasks. We introduce RIFT: RubrIc Failure mode Taxonomy, a taxonomy for systematically characterizing failure modes in rubric composition and design. RIFT consists of eight failure modes organized into three high-level categories: Reliability Failures, Content Validity Failures, and Consequential Validity Failures. RIFT is developed using grounded theory by iteratively annotating rubrics drawn from five diverse data sources spanning general instruction following, code generation, creative writing, and expert-level deep research, until no new failure modes are identified. We evaluate the consistency of the taxonomy by measuring agreement among independent human annotators (87% pairwise agreement, 0.64 average Cohen's kappa) and propose automated rubric quality metrics aligned with human annotations, achieving up to 0.925 F1.\n",
    "tldr": "Grounded-theory taxonomy of 8 rubric-evaluation failure modes across 3 categories (reliability, content validity, consequential validity), developed from rubrics in instruction following, code generation, creative writing, and deep research; paired with automated diagnostic metrics.\n",
    "citation_count": 0,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "evaluation",
      "rubric",
      "grounded-theory",
      "multi-domain"
    ],
    "notes": "Complements SAICA-KG's FailureMode axis from the evaluation/rubric angle rather than the agent-trace angle. Informs how FailureMode tags interact with evaluator quality (meta-evaluation).\n"
  },
  {
    "id": "qiu-2026-prbench-end-end-paper",
    "authors": [
      "Shi Qiu",
      "Jun Deng",
      "Yiwei Deng",
      "Haoran Dong",
      "Jie Fu",
      "Mao Li",
      "Zeyu Li",
      "Zhao Zhang",
      "Huiwen Zheng",
      "Lei Bao",
      "Anqiang Lv",
      "Zihan Mo",
      "Yadi Niu",
      "Yiyang Peng",
      "Yu Tian",
      "Yilin Wang",
      "Ziyu Wang",
      "Ziyu Wang",
      "Jiashen Wei",
      "Liu Wu",
      "Ao Xue",
      "Leyi Yang",
      "Guanglu Yuan",
      "Xia Zhan",
      "Jingjun Zhang",
      "Zifan Zheng",
      "Pengfei Liu",
      "Li Zhen",
      "Kaiyang Li",
      "Qichang Li",
      "Ziheng Zhou",
      "Guokui Nian",
      "Yu-Xiao Xiao",
      "Qing-Hong Cao",
      "L. Dai",
      "Xu Feng",
      "Pengyu Gao",
      "Ying Gu",
      "Chang Liu",
      "Jia Liu",
      "M. Luo",
      "Yan-Qing Ma",
      "Liangchen Peng",
      "Huichao Song",
      "Shufeng Wang",
      "Chenxu Wang",
      "Tao Wang",
      "Yi-Nan Wang",
      "C. Wu",
      "Peng Zhao",
      "H. Zhu"
    ],
    "year": 2026,
    "title": "PRBench: End-to-end Paper Reproduction in Physics Research",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/47471be1d0a60ed97370384746c6bc0accee2473",
    "arxiv_id": "2603.27646",
    "semantic_scholar_id": "47471be1d0a60ed97370384746c6bc0accee2473",
    "abstract": "AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published paper, implement the corresponding algorithms from scratch, and produce quantitative results matching the original publication. Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment. All tasks are contributed by domain experts from over 20 research groups at the School of Physics, Peking University, each grounded in a real published paper and validated through end-to-end reproduction with verified ground-truth results and detailed scoring rubrics. Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution. The best-performing agent, OpenAI Codex powered by GPT-5.3-Codex, achieves a mean overall score of 34%. All agents exhibit a zero end-to-end callback success rate, with particularly poor performance in data accuracy and code correctness. We further identify systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data. Overall, PRBench provides a rigorous benchmark for evaluating progress toward autonomous scientific research.\n",
    "tldr": "This work evaluates a set of coding agents on PRBench and analyzes their capabilities across key dimensions of scientific reasoning and execution, identifying systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data.\n",
    "relevance_tags": [
      "failure-mode-taxonomy",
      "coding-agent",
      "benchmark",
      "guardrails"
    ],
    "notes": "Introduces PRBench, a benchmark of 30 expert-curated end-to-end physics paper-reproduction tasks across 11 subfields, where the agent must comprehend a paper, implement its algorithms from scratch, and produce quantitative results matching the original publication. The best-performing agent (OpenAI Codex powered by GPT-5.3-Codex) achieves only 34% mean overall score, with a zero end-to-end callback success rate. Authors identify systematic failure modes including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data — all directly on-model for SAICA-KG. Relates to SAICA-KG failure modes: fabrication, logic_error, incomplete_execution. Cite when arguing that long-horizon reproducibility is currently the hardest stress test for coding agents.\n"
  },
  {
    "id": "santos-2025-decoding-configuration-ai-coding",
    "authors": [
      "Helio Victor F. Santos",
      "V. Costa",
      "João Eduardo Montandon",
      "Marco Túlio Valente"
    ],
    "year": 2025,
    "title": "Decoding the Configuration of AI Coding Agents: Insights from Claude Code Projects",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/218bfac4d245f01e1d01868e6ae3ae56d741fd27",
    "doi": "10.48550/arXiv.2511.09268",
    "arxiv_id": "2511.09268",
    "semantic_scholar_id": "218bfac4d245f01e1d01868e6ae3ae56d741fd27",
    "abstract": "Agentic code assistants are a new generation of AI systems capable of performing end-to-end software engineering tasks. While these systems promise unprecedented productivity gains, their behavior and effectiveness depend heavily on configuration files that define architectural constraints, coding practices, and tool usage policies. However, little is known about the structure and content of these configuration artifacts. This paper presents an empirical study of the configuration ecosystem of Claude Code, one of the most widely used agentic coding systems. We collected and analyzed 328 configuration files from public Claude Code projects to identify (i) the software engineering concerns and practices they specify and (ii) how these concerns co-occur within individual files. The results highlight the importance of defining a wide range of concerns and practices in agent configuration files, with particular emphasis on specifying the architecture the agent should follow.\n",
    "tldr": "An empirical study of the configuration ecosystem of Claude Code, one of the most widely used agentic coding systems, to identify the software engineering concerns and practices they specify and how these concerns co-occur within individual files.\n",
    "citation_count": 5,
    "relevance_tags": [
      "coding-agent",
      "agentic-ai",
      "empirical"
    ],
    "notes": "Empirical study of 328 configuration files from public Claude Code projects, analyzing the software-engineering concerns they encode (architectural constraints, coding practices, tool usage policies) and how those concerns co-occur. Provides concrete evidence that agent configuration files are a primary supervision substrate in current practice — they are where humans pre-emptively constrain agent behavior. Relates to SAICA-KG failure modes: scope_creep, context_pollution. Useful as a primary cite for the pre_generation TemporalPhase axis (configuration as ex-ante supervision) and as evidence for why a Claude Code-specific Tool node is warranted.\n"
  },
  {
    "id": "shah-2026-characterizing-faults",
    "authors": [
      "Mehil B. Shah",
      "Mohammad Mehdi Morovati",
      "Mohammad Masudur Rahman",
      "Foutse Khomh"
    ],
    "year": 2026,
    "title": "Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2603.06847",
    "arxiv_id": "2603.06847",
    "semantic_scholar_id": "a82a683a61862e69ac7f3b14924d631a6fbd5165",
    "abstract": "Agentic AI systems combine large language model (LLM) reasoning with external tool invocation and long-horizon task execution. We conduct a large-scale empirical study of faults in agentic AI systems, collecting 13,602 issues and pull requests from 40 open-source agentic-AI repositories and applying stratified sampling to select 385 faults for in-depth qualitative analysis. Using grounded theory, we derive taxonomies of fault types, observable symptoms, and root causes. We further apply Apriori-based association rule mining to identify statistically significant relationships among faults, symptoms, and root causes, revealing common fault propagation patterns. We validate the taxonomy through a developer study with 145 practitioners. Our analysis identifies 37 distinct fault types grouped into 13 higher-level fault categories, along with 13 classes of observable symptoms and 12 categories of root causes.\n",
    "tldr": "385-fault grounded-theory study over 40 agentic-AI repos yields 37 fault types in 13 categories, 13 symptom classes, and 12 root-cause categories, validated by a 145-practitioner developer study.\n",
    "citation_count": 1,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "agentic-ai",
      "empirical",
      "external-taxonomy-seed",
      "grounded-theory"
    ],
    "notes": "Primary source for the shah-2026-agentic-faults Taxonomy node in SAICA-KG. Largest single empirical fault characterization of agentic AI to date (385-fault stratified sample, 40 repos, n=145 practitioner validation). Back-test corpus for SAICA-KG's FailureMode MECE claim.\n"
  },
  {
    "id": "song-2025-help-hurdle-rethinking-model",
    "authors": [
      "Wei Song",
      "Haonan Zhong",
      "Ziqi Ding",
      "Jingling Xue",
      "Yuekang Li"
    ],
    "year": 2025,
    "title": "Help or Hurdle? Rethinking Model Context Protocol-Augmented Large Language Models",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/a35799715bfe73519d5be6bfdfaa07014a1f4086",
    "doi": "10.48550/arXiv.2508.12566",
    "arxiv_id": "2508.12566",
    "semantic_scholar_id": "a35799715bfe73519d5be6bfdfaa07014a1f4086",
    "abstract": "The Model Context Protocol (MCP) enables large language models (LLMs) to access external resources on demand. While commonly assumed to enhance performance, how LLMs actually leverage this capability remains poorly understood. We introduce MCPGAUGE, the first comprehensive evaluation framework for probing LLM-MCP interactions along four key dimensions: proactivity (self-initiated tool use), compliance (adherence to tool-use instructions), effectiveness (task performance post-integration), and overhead (computational cost incurred). MCPGAUGE comprises a 160-prompt suite and 25 datasets spanning knowledge comprehension, general reasoning, and code generation. Our large-scale evaluation, spanning six commercial LLMs, 30 MCP tool suites, and both one- and two-turn interaction settings, comprises around 20,000 API calls and over USD 6,000 in computational cost. This comprehensive study reveals four key findings that challenge prevailing assumptions about the effectiveness of MCP integration. These insights highlight critical limitations in current AI-tool integration and position MCPGAUGE as a principled benchmark for advancing controllable, tool-augmented LLMs.\n",
    "tldr": "MCPGAUGE is introduced, the first comprehensive evaluation framework for probing LLM-MCP interactions along four key dimensions: proactivity (self-initiated tool use), compliance, effectiveness, effectiveness, and overhead (computational cost incurred).\n",
    "citation_count": 3,
    "relevance_tags": [
      "MCP",
      "benchmark"
    ],
    "notes": "Introduces MCPGAUGE, the first comprehensive evaluation framework for LLM-MCP interactions along four dimensions: proactivity (self-initiated tool use), compliance (adherence to tool-use instructions), effectiveness (task performance post-integration), and overhead (computational cost). 160- prompt suite across 25 datasets covering knowledge comprehension, general reasoning, and code generation; evaluated on six commercial LLMs and 30 MCP tool suites with ~20,000 API calls. Findings challenge the prevailing assumption that MCP integration unambiguously helps. Relates to SAICA-KG failure modes: scope_creep, context_pollution, incomplete_execution. Cite as evidence that 'tool use' itself is a supervisable behavior, not just a capability.\n"
  },
  {
    "id": "spracklen-2024-we-have-a-package",
    "authors": [
      "Joseph Spracklen",
      "Raveen Wijewickrama",
      "Nazmus Sakib",
      "Anindya Maiti",
      "Bimal Viswanath"
    ],
    "year": 2024,
    "title": "We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs",
    "venue": "USENIX Security",
    "url": "https://www.semanticscholar.org/paper/b76124ef53341af42d6d582ffe340cd0ddb49fc8",
    "abstract": "The reliance of popular programming languages such as Python and JavaScript on centralized package repositories and open-source software, combined with the emergence of code-generating Large Language Models (LLMs), has created a new type of threat to the software supply chain: package hallucinations. These hallucinations, which arise from fact-conflicting errors when generating code using LLMs, represent a novel form of package confusion attack that poses a critical threat to the integrity of the software supply chain. This paper conducts a rigorous and comprehensive evaluation of package hallucinations across different programming languages, settings, and parameters, exploring how a diverse set of models and configurations affect the likelihood of generating erroneous package recommendations and identifying the root causes of this phenomenon. Using 16 popular LLMs for code generation and two unique prompt datasets, we generate 576,000 code samples in two programming languages that we analyze for package hallucinations. Our findings reveal that that the average percentage of hallucinated packages is at least 5.2% for commercial models and 21.7% for open-source models, including a staggering 205,474 unique examples of hallucinated package names, further underscoring the severity and pervasiveness of this threat. To overcome this problem, we implement several hallucination mitigation strategies and show that they are able to significantly reduce the number of package hallucinations while maintaining code quality. Our experiments and findings highlight package hallucinations as a persistent and systemic phenomenon while using state-of-the-art LLMs for code generation, and a significant challenge which deserves the research community's urgent attention.",
    "tldr": "Canonical empirical study of \"slopsquatting\" — package hallucinations by code-generating LLMs. Defines the phenomenon and measures prevalence across commercial and open-source models.\n",
    "citation_count": 55,
    "relevance_tags": [
      "supply-chain",
      "package-hallucination",
      "slopsquatting",
      "failure-mode-evidence"
    ],
    "notes": "Must-cite. Grounds the supply_chain_attack FailureMode and supports SAICA-KG's stance that supply-chain awareness must be built in from node 1.\n"
  },
  {
    "id": "tian-2024-codehalu",
    "authors": [
      "Yuchen Tian",
      "Weixiang Yan",
      "Qian Yang",
      "Xuandong Zhao",
      "Qian Chen",
      "Wen Wang",
      "Ziyang Luo",
      "Lei Ma"
    ],
    "year": 2024,
    "title": "CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification",
    "venue": "AAAI Conference on Artificial Intelligence",
    "url": "https://arxiv.org/abs/2405.00253",
    "doi": "10.1609/aaai.v39i24.34717",
    "arxiv_id": "2405.00253",
    "semantic_scholar_id": "643e596ba6c95e9dfc3bc952f4dc874fc0e9e9fb",
    "abstract": "Large Language Models (LLMs) have made significant progress in code generation, offering developers groundbreaking automated programming support. However, LLMs often generate code that is syntactically correct and even semantically plausible, but may not execute as expected or fulfill specified requirements. This phenomenon of hallucinations in the code domain has not been systematically explored. To advance the community's understanding and research on this issue, we introduce the concept of code hallucinations and propose a classification method for code hallucination based on execution verification. We categorize code hallucinations into four main types: mapping, naming, resource, and logic hallucinations, with each category further divided into different subcategories to understand and address the unique challenges faced by LLMs in code generation with finer granularity. Additionally, we present a dynamic detection algorithm called CodeHalu designed to detect and quantify code hallucinations. We also introduce the CodeHaluEval benchmark, which includes 8,883 samples from 699 tasks, to systematically and quantitatively evaluate code hallucinations. By evaluating 17 popular LLMs using this benchmark, we reveal significant differences in their accuracy and reliability in code generation, offering detailed insights for further improving the code generation capabilities of LLMs.",
    "tldr": "Introduces the concept of code hallucinations and a four-class execution- verification taxonomy (mapping, naming, resource, logic). Provides the CodeHalu detection algorithm and the CodeHaluEval benchmark (8,883 samples over 699 tasks) across 17 LLMs.\n",
    "citation_count": 38,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "hallucination",
      "code-generation",
      "benchmark",
      "execution-based-verification"
    ],
    "notes": "Must-cite. The \"mapping / naming / resource / logic\" axis is a direct predecessor to SAICA-KG's FailureMode partition (fabrication / logic_error / dependency_blindness). Cite as prior art for the fabrication FailureMode and for execution-based detection Tools.\n"
  },
  {
    "id": "twist-2025-library-hallucinations-llms-risk",
    "authors": [
      "Lukas Twist",
      "Jie M. Zhang",
      "Mark Harman",
      "Helen Yannakoudakis"
    ],
    "year": 2025,
    "title": "Library Hallucinations in LLMs: Risk Analysis Grounded in Developer Queries",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/25e6119d93a3f600b1753141a0c2b21a513958fd",
    "doi": "10.48550/arXiv.2509.22202",
    "arxiv_id": "2509.22202",
    "semantic_scholar_id": "25e6119d93a3f600b1753141a0c2b21a513958fd",
    "abstract": "Large language models (LLMs) are increasingly used to generate code, yet they continue to hallucinate, often inventing non-existent libraries. Such library hallucinations are not just benign errors: they can mislead developers, break builds, and expose systems to supply chain threats such as slopsquatting. Despite increasing awareness of these risks, little is known about how real-world prompt variations affect hallucination rates. Therefore, we present the first systematic study of how user-level prompt variations impact library hallucinations in LLM-generated code. We evaluate seven diverse LLMs across two hallucination types: library name hallucinations (invalid imports) and library member hallucinations (invalid calls from valid libraries). We investigate how realistic user language extracted from developer forums and how user errors of varying degrees (one- or multi-character misspellings and completely fake names/members) affect LLM hallucination rates. Our findings reveal systemic vulnerabilities: one-character misspellings in library names trigger hallucinations in up to 26% of tasks, fake library names are accepted in up to 99% of tasks, and time-related prompts lead to hallucinations in up to 84% of tasks. Prompt engineering shows promise for mitigating hallucinations, but remains inconsistent and LLM-dependent. Our results underscore the fragility of LLMs to natural prompt variation and highlight the urgent need for safeguards against library-related hallucinations and their potential exploitation.\n",
    "tldr": "The first systematic study of how user-level prompt variations impact library hallucinations in LLM-generated code reveals systemic vulnerabilities and underscores the fragility of LLMs to natural prompt variation.\n",
    "citation_count": 2,
    "relevance_tags": [
      "supply-chain"
    ],
    "notes": "First systematic study of how user-level prompt variations impact library hallucinations in LLM-generated code. Evaluates seven LLMs across two hallucination types — library-name hallucinations (invalid imports) and library-member hallucinations (invalid calls from valid libraries) — using realistic developer-forum language and controlled misspelling/fake-name perturbations. Striking findings: one-character misspellings trigger hallucinations in up to 26% of tasks, fake library names are accepted in up to 99% of tasks, and time-related prompts hallucinate in up to 84% of tasks (an obsolescence signal). Relates to SAICA-KG failure modes: fabrication, supply_chain_attack, obsolescence, dependency_blindness. The fake-name acceptance rate is a direct slopsquatting attack vector.\n"
  },
  {
    "id": "usman-2017-taxonomy-construction",
    "authors": [
      "Muhammad Usman",
      "Ricardo Britto",
      "Jürgen Börstler",
      "Emilia Mendes"
    ],
    "year": 2017,
    "title": "Taxonomies in Software Engineering: A Systematic Mapping Study and a Revised Taxonomy Development Method",
    "venue": "Information and Software Technology",
    "url": "https://www.sciencedirect.com/science/article/abs/pii/S0950584917300713",
    "tldr": "Canonical mapping study on taxonomy construction methods in SE. Grounds the methodological spine that SAICA-KG's honest-MECE approach inherits.\n",
    "citation_count": 171,
    "relevance_tags": [
      "methodology",
      "taxonomy-construction",
      "faceted-classification"
    ],
    "notes": "Cite as the methodology baseline for SAICA-KG's taxonomy-construction justification. Neutralizes \"your approach is ad hoc\" reviewer critiques.\n"
  },
  {
    "id": "wang-2024-llms-meet-library-evolution",
    "authors": [
      "Chong Wang",
      "Kaifeng Huang",
      "Jian Zhang",
      "et al."
    ],
    "year": 2024,
    "title": "LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion",
    "venue": "ICSE",
    "url": "https://www.semanticscholar.org/paper/llms-meet-library-evolution",
    "abstract": "Large language models (LLMs), pre-trained or fine-tuned on large code corpora, have shown effectiveness in generating code completions. However, in LLM-based code completion, LLMs may struggle to use correct and up-to-date Application Programming Interfaces (APIs) due to the rapid and continuous evolution of libraries. While existing studies have highlighted issues with predicting incorrect APIs, the specific problem of deprecated API usage in LLM-based code completion has not been thoroughly investigated. To address this gap, we conducted the first evaluation study on deprecated API usage in LLM-based code completion. This study involved seven advanced LLMs, 145 API mappings from eight popular Python libraries, and $\\mathbf{2 8, 1 2 5}$ completion prompts. The study results reveal the status quo (i.e., API usage plausibility and deprecated usage rate) of deprecated API and replacing API usage in LLM-based code completion from the perspectives of model, prompt, and library, and indicate the root causes behind. Based on these findings, we propose two lightweight fixing approaches, Replaceapi and InsertPrompt, which can serve as baseline approaches for future research on mitigating deprecated API usage in LLM-based completion. Additionally, we provide implications for future research on integrating library evolution with LLMdriven software development.",
    "tldr": "Canonical study of deprecated-API usage in LLM-based code completion; shows systematic tendency of LLMs to use retired API versions.\n",
    "citation_count": 30,
    "relevance_tags": [
      "obsolescence",
      "deprecated-api",
      "failure-mode-evidence"
    ],
    "notes": "Must-cite. Grounds the obsolescence FailureMode."
  },
  {
    "id": "wang-2025-ai-agentic-programming-survey",
    "authors": [
      "Huanting Wang",
      "Jingzhi Gong",
      "Huawei Zhang",
      "Zheng Wang"
    ],
    "year": 2025,
    "title": "AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2508.11126",
    "doi": "10.48550/arXiv.2508.11126",
    "arxiv_id": "2508.11126",
    "semantic_scholar_id": "280671610",
    "abstract": "AI agentic programming is an emerging paradigm where large language model (LLM)-based coding agents autonomously plan, execute, and interact with tools such as compilers, debuggers, and version control systems. Unlike conventional code generation, these agents decompose goals, coordinate multi-step processes, and adapt based on feedback, reshaping software development practices. This survey provides a timely review of the field, introducing a taxonomy of agent behaviors and system architectures and examining relevant techniques for planning, context management, tool integration, execution monitoring, and benchmarking datasets. We highlight challenges of this fast-moving field and discuss opportunities for building reliable, transparent, and collaborative coding agents.\n",
    "tldr": "AI agentic programming is an emerging paradigm where large language model (LLM)-based coding agents autonomously plan, execute, and interact with tools such as compilers, debuggers, and version control systems. Unlike conventional code generation, these agents decompose goals, coordinate multi-step processes, and adapt based on feedback, reshaping software development practices.\n",
    "citation_count": 17,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "coding-agent",
      "agentic-ai",
      "supervision",
      "benchmark",
      "survey"
    ],
    "notes": "Survey of AI agentic programming: introduces a taxonomy of agent behaviors and system architectures and reviews techniques for planning, context management, tool integration, execution monitoring, and benchmarking datasets. Useful as a coverage map for SAICA-KG's relationship to the wider coding-agent literature; broad rather than deep. Relates to SAICA-KG failure modes: logic_error, context_pollution, incomplete_execution. Cite when situating SAICA-KG against the survey landscape rather than against a primary measurement paper.\n"
  },
  {
    "id": "williams-2025-sscs",
    "authors": [
      "Laurie Williams",
      "Giacomo Benedetti",
      "Sivana Hamer",
      "Ranindya Paramitha",
      "Imranur Rahman",
      "Mahzabin Tamanna",
      "Greg Tystahl",
      "Nusrat Zahan",
      "Patrick Morrison",
      "Yasemin Acar",
      "Michel Cukier",
      "Christian Kastner",
      "Alexandros Kapravelos",
      "Dominik Wermke",
      "William Enck"
    ],
    "year": 2025,
    "title": "Research Directions in Software Supply Chain Security",
    "venue": "ACM Transactions on Software Engineering and Methodology",
    "url": "https://doi.org/10.1145/3714464",
    "doi": "10.1145/3714464",
    "semantic_scholar_id": "461e00ecb0bb5b24b4cb65078a96d0677919f5f3",
    "abstract": "Reusable software libraries, frameworks, and components, such as those provided by open source ecosystems and third-party suppliers, accelerate digital innovation. However, recent years have shown almost exponential growth in attackers leveraging these software artifacts to launch software supply chain attacks. Past well-known software supply chain attacks include the SolarWinds, log4j, and xz utils incidents. Supply chain attacks are considered to have three major attack vectors: through vulnerabilities and malware accidentally or intentionally injected into open source and third-party dependencies/components/containers; by infiltrating the build infrastructure during the build and deployment processes; and through targeted techniques aimed at the humans involved in software development. This article provides an overview of current research efforts to secure the software supply chain and proposes a future research agenda to close software supply chain attack vectors.\n",
    "tldr": "TOSEM roadmap paper on software supply-chain security: organizes attack vectors across dependencies, build infrastructure, and human factors; proposes a future research agenda. 54 citations.\n",
    "citation_count": 54,
    "relevance_tags": [
      "supply-chain-security",
      "software-engineering",
      "roadmap",
      "failure-mode-seed"
    ],
    "notes": "Anchors SAICA-KG's supply_chain_attack FailureMode in the SE-research mainstream. Useful citation backbone for slopsquatting/typosquatting claims and for positioning SAICA-KG against the broader SSC-security literature.\n"
  },
  {
    "id": "xing-2025-mcp-guard-multi-stage",
    "authors": [
      "Wenpeng Xing",
      "Zhong-Kuo Qi",
      "Yupeng Qin",
      "Yilin Li",
      "Caini Chang",
      "Jiahui Yu",
      "Changting Lin",
      "Zhe Xie",
      "Meng Han"
    ],
    "year": 2025,
    "title": "MCP-Guard: A Multi-Stage Defense-in-Depth Framework for Securing Model Context Protocol in Agentic AI",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/8951ab8938bcc4f92e29a6a9f20ae16878090367",
    "arxiv_id": "2508.10991",
    "semantic_scholar_id": "8951ab8938bcc4f92e29a6a9f20ae16878090367",
    "abstract": "While Large Language Models (LLMs) have achieved remarkable performance, they remain vulnerable to jailbreak. The integration of Large Language Models (LLMs) with external tools via protocols such as the Model Context Protocol (MCP) introduces critical security vulnerabilities, including prompt injection, data exfiltration, and other threats. To counter these challenges, we propose MCP-GUARD, a robust, layered defense architecture designed for LLM-tool interactions. MCP-GUARD employs a three-stage detection pipeline that balances efficiency with accuracy: it progresses from lightweight static scanning for overt threats and a deep neural detector for semantic attacks, to our fine-tuned E5-based model which achieves 96.01\\% accuracy in identifying adversarial prompts. Finally, an LLM arbitrator synthesizes these signals to deliver the final decision. To enable rigorous training and evaluation, we introduce MCP-ATTACKBENCH, a comprehensive benchmark comprising 70,448 samples augmented by GPT-4. This benchmark simulates diverse real-world attack vectors that circumvent conventional defenses in the MCP paradigm, thereby laying a solid foundation for future research on securing LLM-tool ecosystems.\n",
    "tldr": "This work proposes MCP-GUARD, a robust, layered defense architecture designed for LLM-tool interactions, and introduces MCP-ATTACKBENCH, a comprehensive benchmark that simulates diverse real-world attack vectors that circumvent conventional defenses in the MCP paradigm.\n",
    "citation_count": 10,
    "relevance_tags": [
      "agentic-ai",
      "MCP",
      "benchmark"
    ],
    "notes": "Proposes MCP-GUARD, a three-stage layered defense for LLM-tool interactions over MCP: lightweight static scanning for overt threats, a deep neural detector for semantic attacks, and a fine-tuned E5-based model achieving 96.01% accuracy on adversarial-prompt detection, with an LLM arbitrator for the final decision. Releases MCP-AttackBench, a 70,448-sample benchmark generated with GPT-4 that simulates real-world attack vectors circumventing conventional defenses. Relates to SAICA-KG failure modes: security_vulnerability, supply_chain_attack. Cite as a layered-defense Tool-node and as one of the better-grounded MCP-defense benchmarks.\n"
  },
  {
    "id": "xu-2025-ckgfuzzer",
    "authors": [
      "Hanxiang Xu",
      "Wei Ma",
      "Ti Zhou",
      "Yanjie Zhao",
      "Kai Chen",
      "Qiang Hu",
      "Yang Liu",
      "Haoyu Wang"
    ],
    "year": 2025,
    "title": "CKGFuzzer: LLM-Based Fuzz Driver Generation Enhanced By Code Knowledge Graph",
    "venue": "ICSE Companion 2025 (IEEE/ACM 47th ICSE-Companion)",
    "url": "https://arxiv.org/abs/2411.11532",
    "doi": "10.1109/ICSE-Companion66252.2025.00079",
    "arxiv_id": "2411.11532",
    "semantic_scholar_id": "23b41a14d8d3496f000bfe01a7f0fdee3fdbc7b4",
    "abstract": "In recent years, the programming capabilities of large language models (LLMs) have garnered significant attention. Fuzz testing, a highly effective technique, plays a key role in enhancing software reliability and detecting vulnerabilities. However, traditional fuzz testing tools rely on manually crafted fuzz drivers, which can limit both testing efficiency and effectiveness. To address this challenge, we propose an automated fuzz testing method driven by a code knowledge graph and powered by an LLM-based intelligent agent system, referred to as CKGFuzzer. We approach fuzz driver creation as a code generation task, leveraging the knowledge graph of the code repository to automate the generation process within the fuzzing loop, while continuously refining both the fuzz driver and input seeds. The code knowledge graph is constructed through interprocedural program analysis, where each node in the graph represents a code entity, such as a function or a file. The knowledge graph-enhanced CKGFuzzer not only effectively resolves compilation errors in fuzz drivers and generates input seeds tailored to specific API usage scenarios, but also analyzes fuzz driver crash reports, assisting developers in improving code quality. By querying the knowledge graph of the code repository and learning from API usage scenarios, we can better identify testing targets and understand the specific purpose of each fuzz driver. We evaluated our approach using eight open-source software projects. The experimental results indicate that CKGFuzzer achieved an average improvement of 8.73% in code coverage compared to state-of-the-art techniques. Additionally, CKGFuzzer reduced the manual review workload in crash case analysis by 84.4% and successfully detected 11 real bugs (including nine previously unreported bugs) across the tested libraries.",
    "tldr": "Automated fuzz-driver generation driven by an interprocedural code knowledge graph of the target repository and an LLM agent. Achieves 8.73% mean coverage improvement over SOTA, reduces crash-triage workload by 84.4%, and finds 11 real bugs (9 previously unreported).\n",
    "citation_count": 16,
    "relevance_tags": [
      "code-knowledge-graph",
      "fuzzing",
      "security",
      "agent-tool",
      "execution-based-verification"
    ],
    "notes": "Must-cite. Concrete precedent for \"code knowledge graph\" as a supervision substrate — directly relevant to SAICA-KG's positioning as a KG over supervision tools and to Tool nodes in the environment locus that use CKG-style interprocedural context.\n"
  },
  {
    "id": "xue-uddin-2025-pagent",
    "authors": [
      "Haoran Xue",
      "Gias Uddin",
      "Song Wang"
    ],
    "year": 2025,
    "title": "PAGENT: Learning to Patch Software Engineering Agents",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2506.17772",
    "doi": "10.48550/arXiv.2506.17772",
    "arxiv_id": "2506.17772",
    "semantic_scholar_id": "4fee1a85d1cb222f3fed5dc1421c8b17f0ae27d2",
    "abstract": "LLM Agents produce patches automatically to resolve an issue. However, they can generate inaccurate patches. Little is known about the root causes behind those failed patches or how those could be fixed. This paper reports an empirical study of the failed patches generated by seven top LLM code agents. We collected 114 issues from the SWE-bench Lite dataset that remained unresolved across the agents. The seven agents produced a total of 769 failed patches for those issues, which we checked with a combination of GPT-4o and manual analysis. We present a taxonomy of the failure reasons across the patches. The taxonomy contains six categories, with several sub-categories under each category. As a first step towards addressing type-related errors, we designed PAGENT (Patch Agent), which utilizes program analysis techniques like CFG creation and exploration to infer type information of a patch. PAGENT fixed 29 of 127 type-related failed patches from the top three agents.\n",
    "tldr": "Empirical study of 769 failed patches from 7 SWE-bench coding agents over 114 unresolved issues; derives a 6-category failure taxonomy and introduces PAGENT, a static-analysis + LLM patch-repair tool that fixes 29/127 type-related failures.\n",
    "citation_count": 4,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "coding-agent",
      "swe-bench",
      "empirical",
      "patch-repair"
    ],
    "notes": "High-signal SWE-bench failure taxonomy; the 6 categories align closely with SAICA-KG's logic_error, fabrication, and dependency_blindness FailureModes. Type-inference angle is a concrete detection-signal source.\n"
  },
  {
    "id": "yan-2025-fault-tolerant-sandboxing",
    "authors": [
      "B. Yan"
    ],
    "year": 2025,
    "title": "Fault-Tolerant Sandboxing for AI Coding Agents: A Transactional Approach to Safe Autonomous Execution",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2512.12806",
    "doi": "10.48550/arXiv.2512.12806",
    "arxiv_id": "2512.12806",
    "semantic_scholar_id": "73b22b88a7b2bb5b4246c6351e3f04303a2ecd52",
    "abstract": "The transition of Large Language Models (LLMs) from passive code generators to autonomous agents introduces significant safety risks, specifically regarding destructive commands and inconsistent system states. Existing commercial solutions often prioritize interactive user safety, enforcing authentication barriers that break the headless loops required for true autonomy. This paper presents a Fault-Tolerant Sandboxing framework designed to mitigate these risks through a policy-based interception layer and a transactional filesystem snapshot mechanism. We hypothesize that wrapping agent actions in atomic transactions can guarantee safety with acceptable latency, outperforming the heavy initialization overhead of containers or the interactive friction of commercial CLIs. We validated this approach by deploying the Minimind-MoE LLM served via nano-vllm on a custom Proxmox-based testbed utilizing EVPN/VXLAN isolation. Experimental results demonstrate a 100\\% interception rate for high-risk commands and a 100\\% success rate in rolling back failed states. Crucially, our prototype incurs only a 14.5\\% performance overhead (approx. 1.8s) per transaction. In contrast, benchmarking against the Gemini CLI sandbox revealed that it requires interactive authentication (\"Sign in\"), rendering it unusable for headless, autonomous agent workflows.",
    "tldr": "Fault-tolerant sandboxing framework wrapping agent actions in atomic transactions over a policy-based interception layer plus a transactional filesystem snapshot mechanism. Demonstrates 100% interception of high-risk commands and 100% rollback of failed states at 14.5% overhead, contrasted against Gemini-CLI's headless-incompatible auth model.\n",
    "citation_count": 1,
    "relevance_tags": [
      "sandboxing",
      "recovery",
      "coding-agent",
      "destructive-action",
      "correction",
      "environment"
    ],
    "notes": "Must-cite. Prototype evidence that recovery ControlParadigm can be made cheap (~1.8s per transaction) for fully-autonomous agents. Directly supports SAICA-KG Tool nodes in the recovery x environment x fully_autonomous cell of the partition cube. Pairs with AgentForge (Kumar 2026) for the sandbox tool cluster.\n"
  },
  {
    "id": "zhang-2025-mcp-security-bench-msb",
    "authors": [
      "Dongsen Zhang",
      "Zekun Li",
      "Xu Luo",
      "Xuannan Liu",
      "Peipei Li",
      "Wenjun Xu"
    ],
    "year": 2025,
    "title": "MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents",
    "venue": "arXiv",
    "url": "https://www.semanticscholar.org/paper/bb33f9e07e661b52a6fade948f71d5a0e0bbc773",
    "doi": "10.48550/arXiv.2510.15994",
    "arxiv_id": "2510.15994",
    "semantic_scholar_id": "bb33f9e07e661b52a6fade948f71d5a0e0bbc773",
    "abstract": "The Model Context Protocol (MCP) standardizes how large language model (LLM) agents discover, describe, and call external tools. While MCP unlocks broad interoperability, it also enlarges the attack surface by making tools first-class, composable objects with natural-language metadata, and standardized I/O. We present MSB (MCP Security Benchmark), the first end-to-end evaluation suite that systematically measures how well LLM agents resist MCP-specific attacks throughout the full tool-use pipeline: task planning, tool invocation, and response handling. MSB contributes: (1) a taxonomy of 12 attacks including name-collision, preference manipulation, prompt injections embedded in tool descriptions, out-of-scope parameter requests, user-impersonating responses, false-error escalation, tool-transfer, retrieval injection, and mixed attacks; (2) an evaluation harness that executes attacks by running real tools (both benign and malicious) via MCP rather than simulation; and (3) a robustness metric that quantifies the trade-off between security and performance: Net Resilient Performance (NRP). We evaluate nine popular LLM agents across 10 domains and 405 tools, producing 2,000 attack instances. Results reveal the effectiveness of attacks against each stage of MCP. Models with stronger performance are more vulnerable to attacks due to their outstanding tool calling and instruction following capabilities. MSB provides a practical baseline for researchers and practitioners to study, compare, and harden MCP agents. Code: https://github.com/dongsenzhang/MSB\n",
    "tldr": "MSB (MCP Security Benchmark), the first end-to-end evaluation suite that systematically measures how well LLM agents resist MCP-specific attacks throughout the full tool-use pipeline: task planning, tool invocation, and response handling, is presented.\n",
    "citation_count": 10,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "agentic-ai",
      "MCP",
      "benchmark"
    ],
    "notes": "Introduces MSB (MCP Security Benchmark), the first end-to-end evaluation suite that measures LLM-agent resistance to MCP-specific attacks across the full tool-use pipeline (task planning, tool invocation, response handling). Contributes a 12-attack taxonomy (name-collision, preference manipulation, prompt injections in tool descriptions, out-of-scope parameter requests, user-impersonating responses, false-error escalation, tool-transfer, retrieval injection, mixed attacks), an evaluation harness that runs real benign and malicious tools via MCP rather than simulation, and the Net Resilient Performance (NRP) robustness metric. Evaluates nine popular LLM agents across 10 domains and 405 tools (2,000 attack instances); finds that stronger models are more vulnerable due to better instruction following. Relates to SAICA-KG failure modes: security_vulnerability, supply_chain_attack, scope_creep. Strong primary cite for the MCP-security FailureMode cluster. Code at github.com/dongsenzhang/MSB.\n"
  },
  {
    "id": "zhang-2025-semanticforge-repository-level-code",
    "authors": [
      "Wuyang Zhang",
      "Chenkai Zhang",
      "Zhen Luo",
      "Jianming Ma",
      "Wangming Yuan",
      "Chuqiao Gu",
      "Chenwei Feng"
    ],
    "year": 2025,
    "title": "SemanticForge: Repository-Level Code Generation through Semantic Knowledge Graphs and Constraint Satisfaction",
    "venue": "INNO-PRESS: Journal of Emerging Applied AI",
    "url": "https://www.semanticscholar.org/paper/7d2789c02f48ebbc7ae052fb6d2b599ee9b88bd6",
    "doi": "10.65563/jeaai.v1i7.65",
    "arxiv_id": "2511.07584",
    "semantic_scholar_id": "7d2789c02f48ebbc7ae052fb6d2b599ee9b88bd6",
    "abstract": "Large language models (LLMs) have transformed software development by enabling automated code generation, yet they frequently suffer from systematic errors that limit practical deployment. We identify two critical failure modes: \\textit{logical hallucination} (incorrect control/data-flow reasoning) and \\textit{schematic hallucination} (type mismatches, signature violations, and architectural inconsistencies). These errors stem from the absence of explicit, queryable representations of repository-wide semantics. This paper presents \\textbf{\\framework}, a novel framework for code generation that addresses these limitations through knowledge graph-guided constraint satisfaction. Our approach proceeds in four integrated stages: (1) constructing heterogeneous repository knowledge graphs that capture both static analysis and dynamic execution traces; (2) learning neural query planners that extract task-relevant context from these graphs; (3) employing satisfiability modulo theories (SMT)-guided beam search to ensure generated code satisfies semantic constraints; and (4) maintaining graph fidelity through continual incremental updates. Our comprehensive evaluation on \\dataset, a curated benchmark of 4,250 repository-level tasks across 50 Python projects, demonstrates significant improvements over state-of-the-art baselines: 49.8\\% Pass@1 (18.1\\% absolute improvement), 52\\% reduction in schematic hallucination, and 31\\% reduction in logical hallucination. Cross-repository generalization analysis shows strong transfer capabilities with only 4.3\\% average performance degradation across architectural patterns. These results establish new benchmarks for repository-level code generation while providing theoretical foundations and practical tools for semantically-aware automated software development. The explicit semantic representation and constraint satisfaction framework introduced in \\framework\\ enables more reliable automated development tools and provides a foundation for future advances in AI-assisted software engineering.\n",
    "tldr": "The explicit semantic representation and constraint satisfaction framework introduced in framework enables more reliable automated development tools and provides a foundation for future advances in AI-assisted software engineering.\n",
    "citation_count": 6,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "benchmark",
      "knowledge-graph"
    ],
    "notes": "Proposes SemanticForge, a repository-level code-generation framework that targets two failure modes the authors name explicitly: 'logical hallucination' (incorrect control/data-flow reasoning, ~ SAICA's logic_error) and 'schematic hallucination' (type mismatches, signature violations, architectural inconsistencies, ~ a mix of logic_error and dependency_blindness). The fix combines a heterogeneous repository knowledge graph (static + dynamic traces), neural query planners, SMT-guided beam search for constraint satisfaction, and incremental graph updates. Reports 49.8% Pass@1 with 52% reduction in schematic hallucination and 31% reduction in logical hallucination on a 4,250-task benchmark across 50 Python projects. Relates to SAICA-KG failure modes: fabrication, logic_error, dependency_blindness. Cite as a peer example of using a knowledge graph as a supervision substrate — directly relevant to SAICA-KG's own architecture argument.\n"
  },
  {
    "id": "zhang-wang-shi-ma-2024-practical-hallucination",
    "authors": [
      "Ziyao Zhang",
      "Chong Wang",
      "Yanlin Wang",
      "Ensheng Shi",
      "Yuchi Ma",
      "Wanjun Zhong",
      "Jiachi Chen",
      "Mingzhi Mao",
      "Zibin Zheng"
    ],
    "year": 2024,
    "title": "LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation",
    "venue": "Proceedings of the ACM on Software Engineering (PACMSE / FSE)",
    "url": "https://arxiv.org/abs/2409.20550",
    "doi": "10.1145/3728894",
    "arxiv_id": "2409.20550",
    "semantic_scholar_id": "75bd806794649ba22d6e994c7ddbdbb169c620fd",
    "abstract": "Code generation aims to automatically generate code from input requirements, significantly enhancing development efficiency. Recent large language models (LLMs) based approaches have shown promising results and revolutionized code generation task. Despite the promising performance, LLMs often generate contents with hallucinations, especially for the code generation scenario requiring the handling of complex contextual dependencies in practical development process. Although previous study has analyzed hallucinations in LLM-powered code generation, the study is limited to standalone function generation. In this paper, we conduct an empirical study to study the phenomena, mechanism, and mitigation of LLM hallucinations within more practical and complex development contexts in repository-level generation scenario. First, we manually examine the code generation results from six mainstream LLMs to establish a hallucination taxonomy of LLM-generated code. Next, we elaborate on the phenomenon of hallucinations, analyze their distribution across different models. We then analyze causes of hallucinations and identify four potential factors contributing to hallucinations. Finally, we propose an RAG-based mitigation method, which demonstrates consistent effectiveness in all studied LLMs.",
    "tldr": "Empirical study of hallucinations in repository-level code generation across six mainstream LLMs. Derives a hallucination taxonomy, distributes phenomena by model, identifies four contributing factors, and proposes an RAG-based mitigation that is effective across the studied models.\n",
    "citation_count": 130,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "hallucination",
      "repository-level",
      "code-generation",
      "mitigation"
    ],
    "notes": "Must-cite. Repository-level framing is closer to SAICA-KG's agent-context scope than function-level hallucination work. Cite as evidence that context_pollution and fabrication both intensify at repo scale and for the RAG mitigation that maps onto context-locus Tools.\n"
  },
  {
    "id": "zhao-2025-hfuzzer-testing-large-language",
    "authors": [
      "Yukai Zhao",
      "Menghan Wu",
      "Xing Hu",
      "Xin Xia"
    ],
    "year": 2025,
    "title": "HFuzzer: Testing Large Language Models for Package Hallucinations via Phrase-based Fuzzing",
    "venue": "International Conference on Automated Software Engineering",
    "url": "https://www.semanticscholar.org/paper/df95b7e84de5ef5bdc2009a0438b8bf32eed4b3e",
    "doi": "10.1109/ASE63991.2025.00225",
    "arxiv_id": "2509.23835",
    "semantic_scholar_id": "df95b7e84de5ef5bdc2009a0438b8bf32eed4b3e",
    "abstract": "Large Language Models (LLMs) are widely used for code generation, but they face critical security risks when applied to practical production due to package hallucinations, in which LLMs recommend non-existent packages. These hallucinations can be exploited in software supply chain attacks, where malicious attackers exploit them to register harmful packages. It is critical to test LLMs for package hallucinations to mitigate package hallucinations and defend against potential attacks. Although researchers have proposed testing frameworks for fact-conflicting hallucinations in natural language generation, there is a lack of research on package hallucinations. To fill this gap, we propose HFuzzer, a novel phrase-based fuzzing framework to test LLMs for package hallucinations. HFuzzer adopts fuzzing technology and guides the model to infer a wider range of reasonable information based on phrases, thereby generating enough and diverse coding tasks. Furthermore, HFuzzer extracts phrases from package information or coding tasks to ensure the relevance of phrases and code, thereby improving the relevance of generated tasks and code. We evaluate HFuzzer on multiple LLMs and find that it triggers package hallucinations across all selected models. Compared to the mutational fuzzing framework, HFuzzer identifies 2.60× more unique hallucinated packages and generates more diverse tasks. Additionally, when testing the model GPT-4o, HFuzzer finds 46 unique hallucinated packages. Further analysis reveals that for GPT-4o, LLMs exhibit package hallucinations not only during code generation but also when assisting with environment configuration.\n",
    "tldr": "H HFuzzer is proposed, a novel phrase-based fuzzing framework to test LLMs for package hallucinations that adopts fuzzing technology and guides the model to infer a wider range of reasonable information based on phrases, thereby generating enough and diverse coding tasks.\n",
    "citation_count": 1,
    "relevance_tags": [
      "supply-chain"
    ],
    "notes": "Proposes HFuzzer, a phrase-based fuzzing framework that systematically tests LLMs for package hallucinations by seeding the fuzzer with phrases drawn from package metadata or coding tasks. Across multiple LLMs HFuzzer triggers package hallucinations in every model tested and identifies 2.60x more unique hallucinated packages than mutational fuzzing baselines; against GPT-4o it surfaces 46 unique hallucinated packages and shows hallucinations occur not just during code generation but also during environment configuration. Relates to SAICA-KG failure modes: supply_chain_attack, fabrication, security_vulnerability. Cite as a fuzzing-style measurement tool that complements static prevalence studies (Spracklen-2024, Krishna-2025).\n"
  },
  {
    "id": "zhu-2025-agent-error-taxonomy",
    "authors": [
      "Zhu",
      "et al."
    ],
    "year": 2025,
    "title": "Where LLM Agents Fail: A Taxonomy of Errors and the AgentErrorBench Dataset",
    "venue": "arXiv",
    "url": "https://arxiv.org/abs/2509.25370",
    "arxiv_id": "2509.25370",
    "abstract": "Large Language Model (LLM) agents, which integrate planning, memory, reflection, and tool-use modules, have shown promise in solving complex, multi-step tasks. Yet their sophisticated architectures amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions, leading to task failure. Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way, and therefore fail to detect these errors accordingly. We address this gap with three contributions. First, we introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations. Second, we construct AgentErrorBench, the first dataset of systematically annotated failure trajectories from ALFWorld, GAIA, and WebShop, grounding error analysis in real-world agent rollouts. Third, we propose AgentDebug, a debugging framework that isolates root-cause failures and provides corrective feedback, enabling agents to recover and iteratively improve. Experiments on AgentErrorBench show that AgentDebug achieves 24% higher all-correct accuracy and 17% higher step accuracy compared to the strongest baseline. Beyond detection, the targeted feedback generated by AgentDebug enables LLM agents to iteratively recover from failures, yielding up to 26% relative improvements in task success across ALFWorld, GAIA, and WebShop. These results establish principled debugging as a pathway to more reliable and adaptive LLM agents. The code and data will be available at https://github.com/ulab-uiuc/AgentDebug",
    "tldr": "AgentErrorTaxonomy organizes LLM agent failures by module (memory, reflection, planning, action, system). Paired with AgentErrorBench dataset.\n",
    "citation_count": 22,
    "relevance_tags": [
      "failure-mode-taxonomy",
      "agent-errors"
    ],
    "notes": "Seed paper for multiple FailureMode prior_work lists. Module-based axis; SAICA-KG absorbs these as multi-valued FailureMode tags rather than as strict partition.\n"
  }
]