CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

Abstract:While autonomous agents show promise for network operations, existing telecom troubleshooting evaluations fail to capture the partial observability and heterogeneity of real-world telecom environments. We introduce CTBench, a public benchmark evaluating agent troubleshooting capabilities in root cause analysis (RCA) and path restoration. Constructed by domain experts, CTBench features rich metadata and golden evidence trajectories, utilizing metrics that assess both final accuracy and diagnostic reasoning. Our evaluation of state-of-the-art agents reveals that while they excel at path endpoint identification, they severely underperform in RCA. Crucially, even when agents produce correct answers, they frequently fail to gather the requisite diagnostic evidence demanded in operational practice. Finally, we demonstrate that increased computational expenditure does not guarantee improved diagnostic competence, highlighting critical gaps in current agent capabilities.

Artificial Intelligence (cs.AI)

arXiv:2608.12002 [cs.AI]

(or arXiv:2608.12002v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2608.12002 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.