Skip to content
aiqbench takeAI Research

aiqbench analysis

The paper tests whether a ReAct-style LLM agent, querying a digital twin's graph through MCP tools, can diagnose telecom and datacenter faults from unstructured operator notes. On nine synthetic scenarios over 360 runs, a frontier model reached 96.7% end-to-end accuracy and a mid-tier model 86.7%, but accuracy fell for all models when evidence had to be combined across several resources. Readers should treat this as a small, synthetic benchmark. The multi-resource drop is the result to watch for any real deployment.
Source coveringarXiv

From Structured Graphs to Unstructured Evidence: Agentic Root Cause Analysis on Digital Twins

arXiv2 min read

arXiv:2601.07342v2 Announce Type: replace Abstract: Root cause analysis over telecom and datacenter infrastructures is commonly addressed by methods that consume structured data, from graph traversal and rule-based correlation to probabilistic and learned models. Yet a digital twin of such an infrastructure cannot model everything at a reasonable cost. Operators therefore record much of what changes as unstructured data attached to the graph, where the discriminating evidence often sits. Methods restricted to structured attributes cannot use it. A diagnosis needs a layer that interprets this data. An agentic layer is a natural candidate, since which fragment matters depends on hypotheses that emerge during the investigation. We demonstrate its feasibility with ReAct, the most widely used agentic architecture. A single LLM interprets the descriptions and explores the graph through a typed tool interface exposed via the Model Context Protocol (MCP). On a benchmark of nine synthetic scenarios covering single-node, multi-node and distractor-heavy evidence (360 runs across four LLMs), a frontier-tier model names the root cause and its type on every run. It reaches 96.7% end-to-end diagnostic accuracy, which also requires the exact supporting evidence. A mid-tier model reaches 86.7%. Accuracy drops for every model when the evidence must be combined across several resources.

Abstract:Root cause analysis over telecom and datacenter infrastructures is commonly addressed by methods that consume structured data, from graph traversal and rule-based correlation to probabilistic and learned models. Yet a digital twin of such an infrastructure cannot model everything at a reasonable cost. Operators therefore record much of what changes as unstructured data attached to the graph, where the discriminating evidence often sits. Methods restricted to structured attributes cannot use it. A diagnosis needs a layer that interprets this data. An agentic layer is a natural candidate, since which fragment matters depends on hypotheses that emerge during the investigation. We demonstrate its feasibility with ReAct, the most widely used agentic architecture. A single LLM interprets the descriptions and explores the graph through a typed tool interface exposed via the Model Context Protocol (MCP). On a benchmark of nine synthetic scenarios covering single-node, multi-node and distractor-heavy evidence (360 runs across four LLMs), a frontier-tier model names the root cause and its type on every run. It reaches 96.7% end-to-end diagnostic accuracy, which also requires the exact supporting evidence. A mid-tier model reaches 86.7%. Accuracy drops for every model when the evidence must be combined across several resources.

Artificial Intelligence (cs.AI)

arXiv:2601.07342 [cs.AI]

(or arXiv:2601.07342v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2601.07342 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Full text under the publisher's open licence · 288 words

Related coverage