What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead
Source
arxiv.org
Author
Haseeb Mohammed Afsar
Date
Context
A directory of AI tools can look healthier when researchers select familiar examples instead of testing a random sample. A study of MCP servers, programs that expose tools to AI applications, randomly selected 400 packages designed to run locally. With no repairs, retries or access credentials, only 195 completed the initial connection exchange, or 48.8%. The same procedure succeeded on 16 of 24 hand-selected servers, or 66.7%.
The comparison shows how selection changes the apparent ease of getting tools running. It measures startup under this particular procedure, rather than the reliability of every MCPThe Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.Full definition → service or whether an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → can successfully use its tools.
Terms in this piece · Glossary
MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
articleCartograph: Federated Tool Discovery with Operator-Attested Retrieval for AI AgentsJustice Owusu Agyemang, Michael Agyare, Kwame Opuni-Boachie Obour Agyekum, Kwame Agyeman-Prempeh Agyekum, Francisca Adoma Acheampong, Jerry John Kponyo
articleCan MCP Clients Decide What to Do After Failure? A Result-Only Actionability AuditRishabh Mehan
articleLaunching MCP Servers on Fly.iofly.io