analiza aiqbench
Composing insight from the full article — this may take a moment.
What Structurally Breaks an Agent? Auditing Difficulty in Stateful Tool-Use Workflows
arXiv:2611.03227v1 Announce Type: new Abstract: An aggregate task score does not distinguish failures of scheduling, state retention, and candidate selection. We develop a small executable instrument that varies dependency depth, candidate count, active state width, and access to current-state snapshots while holding the number of required operations fixed. Two dependency-graph families show that increasing depth at fixed work changes both scheduling freedom and the number of reachable completion states, so these controls are coupled, and we quantify the coupling. The primary study contains 49,152 generated tasks and 393,216 reference-controller rollouts. A separately frozen follow-on design adds 295,936 rollouts covering retention strategy, snapshot cadence, hidden state resampling, recovery, query locality, candidate indexing, and relabeling. Residue caches and precomputed plans solve tasks that defeat limited raw-value caches, and indexed selection removes the serial-scan branching deficit. Hidden resampling imposes an exact success ceiling that additional state tracking cannot overcome. The contribution is an executable diagnostic instrument, analytic calibration, and an audit of its assumptions. A small local CPU study with a 0.5B model completed no workflows, and a 16-episode Copilot-routed fractional factorial is descriptive only. Neither supports general capability claims.
Citește articolul complet pe arXivSubiecte conexe
When Planning Problems Become Hard for LLMs: Separating Structural Complexity from Semantic Interpretability
arXiv · 21 oct. 2026
What Makes Software Issue Resolution Tasks Difficult for Agents?
arXiv · 21 aug. 2026
FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents
arXiv · 11 iun. 2026
QEncodeBench: Can Large Language Models Encode Classical Problems into Verified Quantum Oracles?
arXiv · 18 sept. 2026