aiqbench analysis
MINCE addresses the critical bottleneck of evaluating large language models across diverse variants by using Monte Carlo simulation to drastically reduce benchmark size without relying on learned prediction layers. This approach significantly cuts evaluation time on both GPU and edge NPU hardware while maintaining low accuracy drift, offering a practical solution for scaling model validation in resource-constrained environments.
MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration
arXiv:2606.22826v2 Announce Type: replace Abstract: Evaluating LLMs across many model variants---quantized, fine-tuned, or deployment-specific---requires running large benchmarks repeatedly, a process that can take tens of hours per model on edge hardware such as NPUs. Existing subset selection methods reduce this cost but depend on large calibration pools or learned prediction layers. We introduce MINCE (Monte Carlo Informed N-sizing for Compact Evaluation), which uses Monte Carlo simulation over per-item logs from a small set of calibration models to find the minimum subset size that bounds accuracy drift and then fixes a randomly sampled subset at that size, with no prediction layer needed. MINCE reduces IFEVAL by 54\%, MMLU by 89\%, GSM8K by 70\%, and MMLU-Pro by 88\% with maximum drift $\leq$2.62\,pp on BF16 calibration models. The frozen subsets generalize to held-out GPU models with mean drift $\leq$1.40\,pp and to INT4 NPU models with mean drift of 0.77--3.59\,pp, while delivering evaluation speedups of up to 8.1$\times$ on the GPU models and evaluation speedups of 1.7--3.4$\times$ on the NPU models. The method is robust to calibration pool size and achieves lower drift than tinyBenchmarks (12$\times$ lower on MMLU, 3.3$\times$ on GSM8K) while using 42$\times$ fewer calibration models.
Abstract:Evaluating LLMs across many model variants---quantized, fine-tuned, or deployment-specific---requires running large benchmarks repeatedly, a process that can take tens of hours per model on edge hardware such as NPUs. Existing subset selection methods reduce this cost but depend on large calibration pools or learned prediction layers. We introduce MINCE (Monte Carlo Informed N-sizing for Compact Evaluation), which uses Monte Carlo simulation over per-item logs from a small set of calibration models to find the minimum subset size that bounds accuracy drift and then fixes a randomly sampled subset at that size, with no prediction layer needed. MINCE reduces IFEVAL by 54\%, MMLU by 89\%, GSM8K by 70\%, and MMLU-Pro by 88\% with maximum drift $\leq$2.62\,pp on BF16 calibration models. The frozen subsets generalize to held-out GPU models with mean drift $\leq$1.40\,pp and to INT4 NPU models with mean drift of 0.77--3.59\,pp, while delivering evaluation speedups of up to 8.1$\times$ on the GPU models and evaluation speedups of 1.7--3.4$\times$ on the NPU models. The method is robust to calibration pool size and achieves lower drift than tinyBenchmarks (12$\times$ lower on MMLU, 3.3$\times$ on GSM8K) while using 42$\times$ fewer calibration models.
Artificial Intelligence (cs.AI)
arXiv:2606.22826 [cs.AI]
(or arXiv:2606.22826v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.22826 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 281 words
Related coverage
Kalman Filtering Based Flight Management System Modeling for AAM Aircraft
arXiv · Sep 11, 2026
Knowing When to Defer: Selective Prediction for Responsible Knowledge Tracing
arXiv · Aug 19, 2026
Propagating Elevation-Map Uncertainty Through the Contact Maximum in Closed Form
arXiv · Oct 6, 2026
Real-Time Musculoskeletal Surrogates for Pediatric Cerebral Palsy: a Credibility Pilot
arXiv · Sep 1, 2026