Skip to main content
SAICA-KG

Papers

MCPToolBench++: A Large Scale AI Agent Model Context Protocol MCP Tool Use Benchmark

Shiqing Fan, Xichen Ding, Liang Zhang, Linjian Mo · 2025 · arXiv

Source →arXiv:2508.07575DOI: 10.48550/arXiv.2508.0757523 citations
agentic-aiMCPbenchmark

TL;DR

This work proposes MCPToolBench++, a large-scale, multi-domain AI Agent tool use benchmark, build upon marketplace of over 4k MCP servers from more than 40 categories, collected from the MCP marketplaces and GitHub communities.

Abstract

LLMs'capabilities are enhanced by using function calls to integrate various data sources or API results into the context window. Typical tools include search, web crawlers, maps, financial data, file systems, and browser usage, etc. Integrating these data sources or functions requires a standardized method. The Model Context Protocol (MCP) provides a standardized way to supply context to LLMs. However, the evaluation of LLMs and AI Agents'MCP tool use abilities suffer from several issues. First, there's a lack of comprehensive datasets or benchmarks to evaluate various MCP tools. Second, the diverse formats of response from MCP tool call execution further increase the difficulty of evaluation. Additionally, unlike existing tool-use benchmarks with high success rates in functions like programming and math functions, the success rate of real-world MCP tool is not guaranteed and varies across different MCP servers. Furthermore, the LLMs'context window also limits the number of available tools that can be called in a single run, because the textual descriptions of tool and the parameters have long token length for an LLM to process all at once. To help address the challenges of evaluating LLMs'performance on calling MCP tools, we propose MCPToolBench++, a large-scale, multi-domain AI Agent tool use benchmark. As of July 2025, this benchmark is build upon marketplace of over 4k MCP servers from more than 40 categories, collected from the MCP marketplaces and GitHub communities. The datasets consist of both single-step and multi-step tool calls across different categories. We evaluated SOTA LLMs with agentic abilities on this benchmark and reported the results.

Editorial notes

Introduces MCPToolBench++, a large-scale multi-domain MCP tool use benchmark built on a marketplace of over 4k MCP servers across 40+ categories collected from MCP marketplaces and GitHub. Datasets include single-step and multi-step tool calls evaluated against SOTA agentic LLMs. Notable for explicitly calling out two SAICA-relevant pressures: (a) real-world MCP server success rates are not guaranteed and vary across servers (incomplete_execution), and (b) the LLM context window caps the number of available tools that can be called in a single run because tool descriptions are long-token (context_pollution). Relates to SAICA-KG failure modes: context_pollution, incomplete_execution. Cite as the canonical MCP tool-use evaluation harness.

View raw YAML on GitHub →