analiza aiqbench
This research identifies a critical scalability bottleneck in LLM agent architectures where implicit routing mechanisms fail as irrelevant skill metadata increases, causing significant recall degradation. The study demonstrates that semantic interference from descriptions and instruction conflicts between skill selection and task execution are the primary drivers of this collapse, suggesting that explicit preselection or masking metadata may be necessary for robust scaling. These findings provide essential guidance for designing reliable agent systems capable of managing large libraries of reusable skills without performance decay.
Lost in Implicit Routing: How LLM Agents Fail to Load Relevant Skills at Scale
arXiv:2610.18522v1 Announce Type: new Abstract: Reusable skills let LLM agents load task-specific instructions on demand, but they make execution depend on implicit routing over skill metadata. We test whether this design scales by holding SkillsBench tasks and their golden skills fixed while adding irrelevant skills. We find skill routing collapse: agents increasingly load fewer or no skills, even though the relevant skills remain available. With 1,000 irrelevant skills, loaded-skill recall falls from 73.6% to 27.4% and the zero-load rate rises from 9.3% to 51.6%, whereas explicit selection from the same library retains 73.9% recall. Loading other skills, including plausible alternatives to golden skills, is too rare to explain this decline. Masking only the descriptions of irrelevant skills largely restores skill loading, yet with the metadata intact, explicitly asking agents to plan skill use still yields lower recall than preselection. These findings point to semantic interference from descriptions and instruction interference when skill choice competes with task execution. We also evaluate explicit preselection as a practical mitigation, including its coverage and latency trade-offs. Our analysis informs skill protocols and evaluation for agents with growing skill libraries.
Abstract:Reusable skills let LLM agents load task-specific instructions on demand, but they make execution depend on implicit routing over skill metadata. We test whether this design scales by holding SkillsBench tasks and their golden skills fixed while adding irrelevant skills. We find skill routing collapse: agents increasingly load fewer or no skills, even though the relevant skills remain available. With 1,000 irrelevant skills, loaded-skill recall falls from 73.6% to 27.4% and the zero-load rate rises from 9.3% to 51.6%, whereas explicit selection from the same library retains 73.9% recall. Loading other skills, including plausible alternatives to golden skills, is too rare to explain this decline. Masking only the descriptions of irrelevant skills largely restores skill loading, yet with the metadata intact, explicitly asking agents to plan skill use still yields lower recall than preselection. These findings point to semantic interference from descriptions and instruction interference when skill choice competes with task execution. We also evaluate explicit preselection as a practical mitigation, including its coverage and latency trade-offs. Our analysis informs skill protocols and evaluation for agents with growing skill libraries.
22 pages, including references and appendices. Code: this https URL
Artificial Intelligence (cs.AI)
arXiv:2610.18522 [cs.AI]
(or arXiv:2610.18522v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.18522 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 284 cuvinte
Subiecte conexe
Translating CUDA Tile Operations from Python to Rust Using Agentic AI
NVIDIA · 16 sept. 2026
Presentation: The Right 300 Tokens Beat 100k Noisy Ones: The Architecture of Context Engineering
InfoQ · 14 aug. 2026
Why Software Automation Is Hard
LessWrong · 6 iun. 2026
An Organizational Second Brain: Building an AI That Learns From Experts
Meta Engineering · 2 sept. 2026