aiqbench analysis
This research introduces KCSAT-ML, a dataset derived from Korean national exam mathematics problems that provides per-item difficulty signals grounded in actual human performance rather than synthetic proxies. By pairing this benchmark with the Difficulty-aligned Reasoning Gain (DRG) metric, the study reveals critical flaws in current reasoning models, specifically how test-time scaling often fails to generalize to items humans find difficult and exposes misalignment issues hidden by aggregate accuracy scores. The findings offer a more rigorous framework for evaluating mathematical reasoning capabilities across various vision-language and language models.
KCSAT-ML: Probing Reasoning Models with Nationwide-Cohort Human Difficulty
arXiv:2606.10403v3 Announce Type: replace Abstract: Math reasoning benchmarks have proliferated, yet most lack a per-item difficulty signal grounded in actual human performance. We introduce KCSAT-ML, a decade (2014-2025) of Korean College Scholastic Ability Test (KCSAT; Suneung) mathematics: 664 problems with a 339-item core set carrying official per-item error rates from nationwide cohorts of hundreds of thousands of examinees. We pair the benchmark with Difficulty-aligned Reasoning Gain (DRG): a score-orthogonal metric that asks whether a model's mistakes concentrate on the items humans found hard, or on items humans found easy. Together they expose, across a wide range of VLMs (and LLMs with OCR), three patterns: (i) low-budget accuracy collapses on the high-human-error tail at every model size; (ii) test-time scaling (TTS) raises token use roughly linearly with cohort error rate, while accuracy gains follow a non-monotonic curve; (iii) within a single family, TTS flips between anti-scaling on the hardest items and overthinking on easier ones -- two faces of the same alignment failure. With the proposed DRG metric, we find that models with near-identical accuracy can sit at near-opposite values: one model gets wrong what humans also find hard, while another solves the hardest items yet fails on items humans find easy, which is a critical contrast that aggregate accuracy hides. Our code and dataset builder is fully open-sourced at https://github.com/naver-ai/KCSAT-ML.
Abstract:Math reasoning benchmarks have proliferated, yet most lack a per-item difficulty signal grounded in actual human performance. We introduce KCSAT-ML, a decade (2014-2025) of Korean College Scholastic Ability Test (KCSAT; Suneung) mathematics: 664 problems with a 339-item core set carrying official per-item error rates from nationwide cohorts of hundreds of thousands of examinees. We pair the benchmark with Difficulty-aligned Reasoning Gain (DRG): a score-orthogonal metric that asks whether a model's mistakes concentrate on the items humans found hard, or on items humans found easy. Together they expose, across a wide range of VLMs (and LLMs with OCR), three patterns: (i) low-budget accuracy collapses on the high-human-error tail at every model size; (ii) test-time scaling (TTS) raises token use roughly linearly with cohort error rate, while accuracy gains follow a non-monotonic curve; (iii) within a single family, TTS flips between anti-scaling on the hardest items and overthinking on easier ones -- two faces of the same alignment failure. With the proposed DRG metric, we find that models with near-identical accuracy can sit at near-opposite values: one model gets wrong what humans also find hard, while another solves the hardest items yet fails on items humans find easy, which is a critical contrast that aggregate accuracy hides. Our code and dataset builder is fully open-sourced at this https URL.
24 pages, 14 figures, 13 tables. Accepted to Findings of EMNLP 2026
Computation and Language (cs.CL)
arXiv:2606.10403 [cs.CL]
(or arXiv:2606.10403v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.10403 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 322 words
Related coverage
KCSAT-ML: Probing Reasoning Models with Nationwide-Cohort Human Difficulty
arXiv · Jun 12, 2026
KCSAT-ML: Probing Reasoning Models with Nationwide-Cohort Human Difficulty
arXiv · Jun 10, 2026
Microsoft Discovery Now Generally Available for All Organizations
Quantum Zeitgeist · Jun 2, 2026
A New Era of Discovery: Google Research at I/O 2026
Google Research · May 28, 2026