aiqbench analysis
This study quantifies the performance degradation in enterprise LLM routing as tool catalogs scale from 10 to over 110 agents, revealing significant drops caused by both retrieval failures and inherent model confusion. The authors demonstrate that embedding-based shortlisting effectively mitigates these issues, recovering substantial F1 scores even under real-world traffic conditions with imperfect annotations. For practitioners building large-scale agent systems, this provides critical empirical evidence on the limits of current routing models and viable architectural adjustments to maintain accuracy at scale.
Scaling Enterprise Agent Routing: Degradation, Diagnosis, and Recovery
arXiv:2606.17519v1 Announce Type: cross Abstract: Production LLM assistants route user requests to growing libraries of specialized tools, but how does routing accuracy degrade as the catalog scales? We study single-step routing on a 110-agent, 584-tool catalog from a deployed enterprise productivity assistant, evaluating three frontier models from 10 to 110 agents. Routing F1 on under-specified requests drops 16--23 percentage points across models. An oracle analysis decomposes the degradation into a \emph{retrieval} gap (the model cannot surface the right tool) and a \emph{confusion} gap (even with perfect retrieval, the oracle ceiling drops 10pp). Embedding-based shortlisting recovers +10--11pp F1 at full scale across all three models and two providers. A production annotation study (1,435 human-labeled utterances, three annotators) confirms the recovery on real traffic at +10--17pp despite 10--15pp lower absolute performance.
Abstract:Production LLM assistants route user requests to growing libraries of specialized tools, but how does routing accuracy degrade as the catalog scales? We study single-step routing on a 110-agent, 584-tool catalog from a deployed enterprise productivity assistant, evaluating three frontier models from 10 to 110 agents. Routing F1 on under-specified requests drops 16--23 percentage points across models. An oracle analysis decomposes the degradation into a \emph{retrieval} gap (the model cannot surface the right tool) and a \emph{confusion} gap (even with perfect retrieval, the oracle ceiling drops 10pp). Embedding-based shortlisting recovers +10--11pp F1 at full scale across all three models and two providers. A production annotation study (1,435 human-labeled utterances, three annotators) confirms the recovery on real traffic at +10--17pp despite 10--15pp lower absolute performance.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.17519 [cs.CL]
(or arXiv:2606.17519v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.17519 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 222 words
Related coverage
APEX: A Network-Native Time-Series Foundation Model for Forecasting and Anomaly Detection for Wireless Edge Operations
arXiv · Jun 12, 2026
Wi-Fi 8: The next evolution in enterprise connectivity
Cisco Blogs · Oct 20, 2026
Scalable On-Hardware Training of Quantum Neural Networks and Application to Clinical Data Imputation
arXiv · Jun 3, 2026
From PDFs to insights: Architecting an intelligent document processing pipeline with AWS generative AI services
AWS · Jun 12, 2026