analiza aiqbench
This paper is squarely about AI: it evaluates LLM agents on experimental reasoning in a synthetic world with known ground truth. Its main contribution is a benchmark that separates experiment selection, prediction, diagnosis, and intervention, showing that these abilities diverge sharply (diagnosis varied about 7x across models versus 1.4x for prediction). The practical takeaway for AI readers is that outcome-only evaluations can hide agents that rescue systems for the wrong reasons, and that more exploration does not reliably improve results. Because it is a workshop, non-archival preprint, its findings should be treated as early evidence rather than settled results.
LUCA: A Synthetic World for Experimental Reasoning
arXiv:2611.03100v1 Announce Type: new Abstract: An experimental agent can succeed for the wrong reasons. Evaluations, however, often focus on the outcome, not the experiments, evidence, and inferences that produced it. We introduce LUCA, a configurable synthetic world for evaluating this process under fully known ground truth. Agents learn unknown regulatory systems through budgeted experimentation, then predict unseen interventions, diagnose hidden faults, and attempt rescues. This setup allows us to evaluate experiment selection, prediction, diagnosis, and intervention separately. Across seven frontier LLMs and three baselines, these capabilities dissociate sharply. Agents frequently rescue malfunctioning systems while diagnosing the wrong fault; diagnosis varies 7x across models while prediction varies only 1.4x. Models also adopt surprisingly different experimental strategies, and greater exploration does not reliably yield better performance. In one failure mode, agents often perform the experiment needed to distinguish competing causes, but still fail to use the resulting evidence. LUCA turns these hidden failures into measurable behaviours, providing a controlled environment to study where experimental agents succeed and fail. Code: https://github.com/RelationRx/luca.
Abstract:An experimental agent can succeed for the wrong reasons. Evaluations, however, often focus on the outcome, not the experiments, evidence, and inferences that produced it. We introduce LUCA, a configurable synthetic world for evaluating this process under fully known ground truth. Agents learn unknown regulatory systems through budgeted experimentation, then predict unseen interventions, diagnose hidden faults, and attempt rescues. This setup allows us to evaluate experiment selection, prediction, diagnosis, and intervention separately. Across seven frontier LLMs and three baselines, these capabilities dissociate sharply. Agents frequently rescue malfunctioning systems while diagnosing the wrong fault; diagnosis varies 7x across models while prediction varies only 1.4x. Models also adopt surprisingly different experimental strategies, and greater exploration does not reliably yield better performance. In one failure mode, agents often perform the experiment needed to distinguish competing causes, but still fail to use the resulting evidence. LUCA turns these hidden failures into measurable behaviours, providing a controlled environment to study where experimental agents succeed and fail. Code: this https URL.
20 pages (9 pages main text, remainder appendix), 6 figures, 14 tables. NeurIPS 2026 Workshop on Evaluation of Interactive Agents; non-archival
Artificial Intelligence (cs.AI)
arXiv:2611.03100 [cs.AI]
(or arXiv:2611.03100v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2611.03100 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 280 cuvinte
Subiecte conexe
From Simulation to Experiment: Benchmarking Machine-Learning Models for Quantum-Dot State Recognition
arXiv · 27 oct. 2026
Rice Builds AI to Tackle Quantum Optimization Bottlenecks
Quantum Zeitgeist · 25 iul. 2026
From Behavior to Provenance: Attributing Tabular Foundation Models to Synthetic Pretraining Data
arXiv · 5 oct. 2026
Robustness of Similarity-based Positional Encoding Under Rotations: Theoretical Analysis and Experimental Validation
arXiv · 17 iun. 2026