aiqbench analysis
This research exposes critical flaws in how large language models handle quantitative reasoning within environmental science, revealing that standard multiple-choice benchmarks artificially inflate accuracy scores by over 12 percentage points. The study demonstrates that even advanced models often fail to consistently apply known physical formulas and constraints during multi-step computations, instead reverting to generic patterns when specific conditions invalidate familiar methods. Consequently, the authors argue that expert oversight remains essential for validating model outputs in high-stakes scientific applications where calculation processes must be transparent and reliable.
Execution-grounded evaluation reveals hidden failures in language-model calculations for environmental science
arXiv:2608.18726v1 Announce Type: new Abstract: Large language models are increasingly used for quantitative work in the environmental sciences, yet existing evaluations score only final answers, leaving calculation process unobserved. Here we introduce AtmosCoder-Bench, an execution-grounded benchmark that makes the calculation process visible. Built through a transferable semi-automated pipeline (436 problems, 3,910 variants, 7,029 graded quantities), every problem is validated to be unambiguous and human-solvable, with uniquely verifiable answers. We find that (i) multiple-choice formats inflate measured accuracy by at least 12 percentage points; (ii) many failures arise not from missing knowledge but from models failing to apply known formulas and constraints consistently throughout multi-step computation; and (iii) even frontier models remain weak when task-specific conditions invalidate familiar methods, often reverting to canonical solution patterns rather than adapting methods to the relevant physical regime, leaving expert oversight essential.
Abstract:Large language models are increasingly used for quantitative work in the environmental sciences, yet existing evaluations score only final answers, leaving calculation process unobserved. Here we introduce AtmosCoder-Bench, an execution-grounded benchmark that makes the calculation process visible. Built through a transferable semi-automated pipeline (436 problems, 3,910 variants, 7,029 graded quantities), every problem is validated to be unambiguous and human-solvable, with uniquely verifiable answers. We find that (i) multiple-choice formats inflate measured accuracy by at least 12 percentage points; (ii) many failures arise not from missing knowledge but from models failing to apply known formulas and constraints consistently throughout multi-step computation; and (iii) even frontier models remain weak when task-specific conditions invalidate familiar methods, often reverting to canonical solution patterns rather than adapting methods to the relevant physical regime, leaving expert oversight essential.
29 pages, 4 figures, 2 tables, plus supplementary materials. Maohao Ran and Chendong Ma contributed equally. Corresponding author: Jun Song (junsong@hkbu.this http URL). Code: this https URL
Computation and Language (cs.CL)
arXiv:2608.18726 [cs.CL]
(or arXiv:2608.18726v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.18726 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 178 words
Related coverage
Always ask your agent to be honest
LessWrong · Sep 16, 2026
[AINews] not much happened today
Latent Space · Sep 18, 2026
Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs
arXiv · Jun 9, 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · May 19, 2026