aiqbench analysis
StudyBench introduces a rigorous physics benchmark to quantify how efficiently self-evolution methods convert raw training data into transferable problem-solving skills, specifically distinguishing between textbook absorption and Olympiad-level application. The analysis reveals a critical "Guidance Gap" where current methods fail to fully utilize available material compared to in-context guidance, alongside a universal compute saturation limit that points to algorithmic inefficiencies rather than resource scarcity. This work provides an essential diagnostic tool for researchers aiming to optimize autonomous learning pipelines without relying on brute-force scaling.
StudyBench: Can Self-Evolution Squeeze Textbooks for Olympiad Capability?
arXiv:2609.00787v1 Announce Type: new Abstract: Humans need to study only a handful of well-written textbooks to master a discipline and attempt its hardest problems. We argue that an ideal self-evolution method should share the same property, that is autonomously learning from raw training material for transferable problem-solving capability. However, we still lack a direct measurement for it. We introduce StudyBench, a controlled physics benchmark that directly measures how efficiently a self-evolution method converts training material into capability. We organise the test set into an Application Set, consisting of difficult textbook problems and evaluating absorption ability, and a Transfer Set, consisting of olympiad-level problems and evaluating transfer ability. Benchmarking representative self-evolution methods across three base models, we find that improvements on the Application Set rarely translate to the harder Transfer Set. A guidance ablation exposes a Guidance Gap: even the strongest method closes only a small fraction of what the same material unlocks when supplied as in-context guidance. Besides, every method hits a Compute Plateau, saturating well before exhausting its compute budget. The remaining gap is therefore a method problem rather than a data or compute problem. By offering a clean and controlled benchmark, StudyBench turns self-evolution progress from an open-ended pursuit into a measurable target for future research. Our code is released at https://github.com/thunlp/StudyBench.
Title:StudyBench: Can Self-Evolution Squeeze Textbooks for Olympiad Capability?
Abstract:Humans need to study only a handful of well-written textbooks to master a discipline and attempt its hardest problems. We argue that an ideal self-evolution method should share the same property, that is autonomously learning from raw training material for transferable problem-solving capability. However, we still lack a direct measurement for it. We introduce StudyBench, a controlled physics benchmark that directly measures how efficiently a self-evolution method converts training material into capability. We organise the test set into an Application Set, consisting of difficult textbook problems and evaluating absorption ability, and a Transfer Set, consisting of olympiad-level problems and evaluating transfer ability. Benchmarking representative self-evolution methods across three base models, we find that improvements on the Application Set rarely translate to the harder Transfer Set. A guidance ablation exposes a Guidance Gap: even the strongest method closes only a small fraction of what the same material unlocks when supplied as in-context guidance. Besides, every method hits a Compute Plateau, saturating well before exhausting its compute budget. The remaining gap is therefore a method problem rather than a data or compute problem. By offering a clean and controlled benchmark, StudyBench turns self-evolution progress from an open-ended pursuit into a measurable target for future research. Our code is released at this https URL.
Artificial Intelligence (cs.AI)
arXiv:2609.00787 [cs.AI]
(or arXiv:2609.00787v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.00787 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 313 words
Related coverage
When they can perform a task, AIs are much cheaper than humans
LessWrong · Sep 27, 2026
Against Export Controls (and China Threat Models)
LessWrong · Sep 24, 2026
Presentation: From Hype to Strong Foundations: What the Rise, Fall and Resurgence of Agents Can Teach Us About Outlasting the Cycle
InfoQ · Jun 17, 2026
Synthesis Superintelligence: from Semiconductors to Superconductors — Periodic Labs’ Liam Fedus and Ekin Dogus Cubuk
Latent Space · Oct 8, 2026