analiza aiqbench
RankLLM addresses a critical gap in current LLM evaluation by introducing a framework that quantifies question difficulty alongside model competency, enabling more granular performance comparisons. By utilizing bidirectional score propagation between models and questions, the method achieves high agreement with human judgments while offering the stability and efficiency required for large-scale benchmarking. This approach represents a significant methodological advance for researchers seeking to move beyond standardized metrics that fail to distinguish between model capabilities and inherent question hardness.
RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty
arXiv:2602.12424v2 Announce Type: replace Abstract: Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field. However, existing benchmarks fail to differentiate question difficulty, limiting their ability to effectively distinguish models' capabilities. To address this limitation, we propose RankLLM, a novel framework designed to quantify both question difficulty and model competency. RankLLM introduces difficulty as the primary criterion for differentiation, enabling a more fine-grained evaluation of LLM capabilities. RankLLM's core mechanism facilitates bidirectional score propagation between models and questions. The core intuition of RankLLM is that a model earns a competency score when it correctly answers a question, while a question's difficulty score increases when it challenges a model. Using this framework, we evaluate 30 models on 35,550 questions across multiple domains. RankLLM achieves 90% agreement with human judgments and consistently outperforms strong baselines such as IRT. It also exhibits strong stability, fast convergence, and high computational efficiency, making it a practical solution for large-scale, difficulty-aware LLM evaluation.
Abstract:Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field. However, existing benchmarks fail to differentiate question difficulty, limiting their ability to effectively distinguish models' capabilities. To address this limitation, we propose RankLLM, a novel framework designed to quantify both question difficulty and model competency. RankLLM introduces difficulty as the primary criterion for differentiation, enabling a more fine-grained evaluation of LLM capabilities. RankLLM's core mechanism facilitates bidirectional score propagation between models and questions. The core intuition of RankLLM is that a model earns a competency score when it correctly answers a question, while a question's difficulty score increases when it challenges a model. Using this framework, we evaluate 30 models on 35,550 questions across multiple domains. RankLLM achieves 90% agreement with human judgments and consistently outperforms strong baselines such as IRT. It also exhibits strong stability, fast convergence, and high computational efficiency, making it a practical solution for large-scale, difficulty-aware LLM evaluation.
32 pages, 9 figures. Accepted by ICLR 2026
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2602.12424 [cs.CL]
(or arXiv:2602.12424v2 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 190 cuvinte
Subiecte conexe
Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
arXiv · 30 sept. 2026
LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models
arXiv · 1 sept. 2026
VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models
arXiv · 3 iun. 2026
Evaluation of Blood Vessel Segmentation Methods on Hard-to-Detect Vascular Structures
arXiv · 28 iul. 2026