aiqbench analysis
FrontierMath Erdős (FME) establishes a rigorous benchmark using 68 open conjectures from the Erdős database, requiring AI systems to prove or disprove them within Lean under strict budget constraints. The study reveals that current models, including GPT-6 Astra, struggle significantly with this level of mathematical rigor, scoring near zero compared to human performance on these specific problems. This work highlights the substantial gap between general-purpose reasoning and the systematic resolution of deep mathematical conjectures in formal proof environments.
FrontierMath Erd\H{o}s
arXiv:2609.25050v1 Announce Type: cross Abstract: We introduce FrontierMath Erd\H{o}s (FME), a benchmark of 68 Erd\H{o}s problems that are open as of August 2026. To solve a task in FME, AI systems must resolve (prove or disprove) one of the 68 conjectures in the proof assistant Lean. Our 68 problems were selected by the second author among 652 open problems on erdosproblems.com for their mathematical interest and difficulty. AIs have recently resolved several open problems in mathematics, but these demonstrations fall short of a systematic study of AI capabilities. FME evaluates every AI model on the same fixed problems, autonomously and under the same budget. We evaluated five AIs with a budget of \$300 per problem. One (GPT-6 Astra) scored 3%, and all others scored 0%.
Abstract:We introduce FrontierMath Erdős (FME), a benchmark of 68 Erdős problems that are open as of August 2026. To solve a task in FME, AI systems must resolve (prove or disprove) one of the 68 conjectures in the proof assistant Lean. Our 68 problems were selected by the second author among 652 open problems on this http URL for their mathematical interest and difficulty. AIs have recently resolved several open problems in mathematics, but these demonstrations fall short of a systematic study of AI capabilities. FME evaluates every AI model on the same fixed problems, autonomously and under the same budget. We evaluated five AIs with a budget of \$300 per problem. One (GPT-6 Astra) scored 3%, and all others scored 0%.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2609.25050 [cs.CL]
(or arXiv:2609.25050v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.25050 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 218 words
