aiqbench analysis
This paper introduces BenchHub, a framework that deconstructs Large Language Model benchmarks into their constituent instance attributes to reveal hidden composition biases and performance variations masked by aggregate scores. By integrating over 50 existing benchmarks, it provides researchers with tools to inspect coverage, compare datasets transparently, and construct controllable evaluation sets for more accurate model selection. The work fundamentally shifts the paradigm of AI evaluation from treating benchmarks as monolithic units to analyzing them as heterogeneous distributions that require explicit inspection.
Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub
arXiv:2506.00482v2 Announce Type: replace-cross Abstract: LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills, formats, and contexts. As a result, aggregate benchmark scores can conflate model capability with benchmark composition, obscuring what existing benchmarks actually cover. We introduce BenchHub, a composition-aware evaluation framework that represents benchmarks as distributions over instance-level attributes. BenchHub integrates 54 benchmarks comprising 839K samples across 10 languages. It enables researchers to inspect benchmark contents, uncover reusable coverage hidden behind benchmark names, transparently compare benchmarks, and construct controllable evaluation sets for application-aligned model selection. Using BenchHub, we show that similarly motivated benchmarks can differ substantially in internal composition, existing benchmarks often contain reusable coverage beyond their stated purposes, models exhibit fine-grained category-level performance variation hidden by aggregate scores, and model rankings can shift under different reweighting and resampling configurations. Our results motivate evaluation practices that make benchmark composition explicit, inspectable, and controllable.
Abstract:LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills, formats, and contexts. As a result, aggregate benchmark scores can conflate model capability with benchmark composition, obscuring what existing benchmarks actually cover. We introduce BenchHub, a composition-aware evaluation framework that represents benchmarks as distributions over instance-level attributes. BenchHub integrates 54 benchmarks comprising 839K samples across 10 languages. It enables researchers to inspect benchmark contents, uncover reusable coverage hidden behind benchmark names, transparently compare benchmarks, and construct controllable evaluation sets for application-aligned model selection. Using BenchHub, we show that similarly motivated benchmarks can differ substantially in internal composition, existing benchmarks often contain reusable coverage beyond their stated purposes, models exhibit fine-grained category-level performance variation hidden by aggregate scores, and model rankings can shift under different reweighting and resampling configurations. Our results motivate evaluation practices that make benchmark composition explicit, inspectable, and controllable.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2506.00482 [cs.LG]
(or arXiv:2506.00482v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2506.00482 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 250 words
Related coverage
VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric
arXiv · Oct 15, 2026
Scalable Question-Centric Text-to-Image Evaluation: Reliable Ranking, Fine-Grained Diagnosis, and Cost-Aware Routing
arXiv · Aug 27, 2026
Researchers Measure Quantum Processor Speed Using New Benchmark
Quantum Zeitgeist · Sep 11, 2026
Grounded Skill Synthesis from Code at Scale for Agentic Intelligence
arXiv · Sep 9, 2026