analiza aiqbench
This research introduces a novel method that uses large language models to generate and validate natural-language explanations for why specific questions are difficult, moving beyond simple difficulty scores to uncover causal factors in reasoning tasks. By applying Item Response Theory and validating hypotheses on held-out data across mathematical, logical, and commonsense domains, the study demonstrates that these interpretable features can predict model performance better than black-box regressors and improve existing evaluation metrics. This work significantly advances the field of AI benchmarking by transforming opaque difficulty metrics into actionable insights for dataset construction and model assessment.
What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language
arXiv:2610.01627v1 Announce Type: cross Abstract: Difficulty is one of the most fundamental properties of a question: it determines whether the question can meaningfully discriminate between models of differing ability. Although a variety of methods can now estimate or predict difficulty automatically, they yield only a single descriptive number, with no account of the underlying factors that make a question difficult in the first place. In this work, we propose a data-driven approach that automatically generates and validates natural-language hypotheses explaining what makes one question harder than another. We first estimate each item's difficulty from the responses of a large pool of LLMs using Item Response Theory. We then sample contrasting sets of easy and hard questions and prompt an LLM to propose candidate explanations of the difference, which are subsequently validated and selected on held-out questions. Experimental results across three datasets spanning mathematical, logical, and commonsense reasoning show that our method produces interpretable and predictive hypotheses. On their own, they predict the difficulty of unseen questions competitively with, or better than, advanced black-box difficulty regressors; used as additional features, they further improve those regressors, implying that they discover difficulty signals that existing models fail to capture. Moreover, we demonstrate that editing questions according to a hypothesis can shift their measured difficulty in the expected direction, indicating that the discovered hypotheses are causally valid difficulty factors rather than post-hoc descriptions. Our approach thus turns a purely descriptive difficulty score into actionable statements.
Title:What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language
Abstract:Difficulty is one of the most fundamental properties of a question: it determines whether the question can meaningfully discriminate between models of differing ability. Although a variety of methods can now estimate or predict difficulty automatically, they yield only a single descriptive number, with no account of the underlying factors that make a question difficult in the first place. In this work, we propose a data-driven approach that automatically generates and validates natural-language hypotheses explaining what makes one question harder than another. We first estimate each item's difficulty from the responses of a large pool of LLMs using Item Response Theory. We then sample contrasting sets of easy and hard questions and prompt an LLM to propose candidate explanations of the difference, which are subsequently validated and selected on held-out questions. Experimental results across three datasets spanning mathematical, logical, and commonsense reasoning show that our method produces interpretable and predictive hypotheses. On their own, they predict the difficulty of unseen questions competitively with, or better than, advanced black-box difficulty regressors; used as additional features, they further improve those regressors, implying that they discover difficulty signals that existing models fail to capture. Moreover, we demonstrate that editing questions according to a hypothesis can shift their measured difficulty in the expected direction, indicating that the discovered hypotheses are causally valid difficulty factors rather than post-hoc descriptions. Our approach thus turns a purely descriptive difficulty score into actionable statements.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2610.01627 [cs.CL]
(or arXiv:2610.01627v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.01627 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 344 cuvinte
Subiecte conexe
Lattice-Based Cryptography Explained
Quantum Zeitgeist · 15 aug. 2026
Deconfusing Corrigibility
AI Alignment Forum · 17 oct. 2026
Biological Superintelligence
LessWrong · 30 iul. 2026
The Alignment Problem: Computer Science Meets Analytic Philosophy
AI Alignment Forum · 18 oct. 2026