aiqbench analysis
The paper introduces SlowLab, a benchmark that tests whether language-model agents can run a year-long, irreversible experimental campaign in simulated greenhouses. In a preregistered study, three low-cost agents ended near a standard grower setting and below a Bayesian-optimisation baseline, and in-season measurements did not improve any method's results. For readers, the useful signal is that current agents struggle with slow, costly, hidden-information experiments, and that the bottleneck appears to lie in their initial hypotheses and experiment choices rather than in how they analyse evidence.
SlowLab: Benchmarking Autonomous AI Agents on a Year of Experiments They Cannot Undo
arXiv:2610.34644v1 Announce Type: new Abstract: AI agents are increasingly proposed as autonomous experimenters, yet the benchmarks that test them mostly offer cheap experiments that answer at once. Experiments in physical facilities are different: they take months, occupy scarce equipment and cannot be undone. We present SlowLab, a benchmark in which an agent runs a one-year campaign in four simulated greenhouse compartments on a site whose crop, facility and prices are hidden. The agent decides when to plant which climate policy, whether to stop a crop and which measurements to read while crops grow, and at the end recommends one policy, which a private evaluator scores in weather the agent never saw. In a preregistered study on 48 unseen sites, three low-cost language-model agents ended near a standard grower setting and clearly below a Bayesian-optimisation baseline that starts from that setting. They ran the facility as two synchronous batches, so measurements taken during the season never changed what they planted, and these measurements improved no method's result. Letting other methods re-read the evidence a campaign had collected did not yield significantly better recommendations, which points to where the models start and which experiments they try. Code is released under the MIT license.
Abstract:AI agents are increasingly proposed as autonomous experimenters, yet the benchmarks that test them mostly offer cheap experiments that answer at once. Experiments in physical facilities are different: they take months, occupy scarce equipment and cannot be undone. We present SlowLab, a benchmark in which an agent runs a one-year campaign in four simulated greenhouse compartments on a site whose crop, facility and prices are hidden. The agent decides when to plant which climate policy, whether to stop a crop and which measurements to read while crops grow, and at the end recommends one policy, which a private evaluator scores in weather the agent never saw. In a preregistered study on 48 unseen sites, three low-cost language-model agents ended near a standard grower setting and clearly below a Bayesian-optimisation baseline that starts from that setting. They ran the facility as two synchronous batches, so measurements taken during the season never changed what they planted, and these measurements improved no method's result. Letting other methods re-read the evidence a campaign had collected did not yield significantly better recommendations, which points to where the models start and which experiments they try. Code is released under the MIT license.
16 pages (9 main text), 7 figures, 5 tables. Code: this https URL
Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
arXiv:2610.34644 [cs.AI]
(or arXiv:2610.34644v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.34644 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 306 words
Related coverage
Towards Robust Long-Lived Agents: Benchmarking Experience Lifecycles in Evolving Environments
arXiv · Nov 5, 2026
Are Benchmarks Reliable? Toward Structural Diagnosis via Sample-Level Capability Boundaries
arXiv · Sep 29, 2026
FinEvolveBench: A Benchmark for Self-Evolving Agents on Low-Repetition Tasks with Implicit Rewards
arXiv · Sep 30, 2026
From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs
arXiv · Aug 6, 2026