Scaling LLM Agent Tool Access: The MCP Gateway

Abstract:LLM agents increasingly rely on tool calling to complete tasks. Model Context Protocol (MCP) standardizes tool calling between LLM agents and tool providers. It reduces integration cost and drives rapid growth in tool scale. In production deployments, tool access becomes difficult to scale. On the tool provider side, legacy services are not directly callable through MCP; the rapid protocol development creates ongoing compatibility cost. On the agent side, the number of mounted tools is limited by the LLM context window and inference overhead; mounting a large tool set increases token usage and inference latency and can reduce task success rate. Moreover, for stateful MCP backends with multiple replicas, preserving session affinity increases client-side complexity. We present a cloud-scale MCP Gateway system. It breaks the direct-connect model on the data plane and offloads legacy service integration, consolidating incompatible MCP variants, access control, tool recommendation, and session-aware routing at the gateway. The conversion module supports 315 OpenAPI services in the cloud and integrates MCP transport variants. Session-aware routing decouples the scalability of the gateway layer and the MCP service layer. The hybrid retrieval engine sustains $\sim$98\% Top-15 recall; it scales agent tool access to 3{,}000+ tools with improved tool selection accuracy. We also share the lessons learned from deploying the gateway system in production.

Accepted at the 2026 ACM SIGOPS Annual Technical Conference (SIGOPS ATC 2026). Compared with the ATC version, this version adds supplementary appendix material and corrects an error in Figure 9. See the statement for details

Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Networking and Internet Architecture (cs.NI)

arXiv:2607.15593 [cs.DC]

(or arXiv:2607.15593v2 [cs.DC] for this version)

https://doi.org/10.1145/3828523.3857007 DOI(s) linking to related resources