aiqbench analysis
This research exposes the "illusion of equivalency" in quantized large language models, demonstrating that standard metrics like accuracy and perplexity fail to capture significant behavioral shifts between base and compressed variants. By introducing a new decision-level metric called Correctness Agreement, the study reveals that attention weight structures, particularly query and key projections, degrade differently at low bit widths than traditional evaluations suggest. These findings are critical for developers deploying models on resource-constrained devices, urging a move beyond standard benchmarks to assess true functional fidelity in quantized systems.
The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
arXiv:2607.08734v2 Announce Type: replace Abstract: Post-Training Quantization has become widely used to compress large language models to make them deployable on resource-constrained devices. However, the evaluation of quantization methods mainly uses accuracy and perplexity, which cannot capture the behavioral changes in the quantized variants. In this work, we propose Correctness Agreement, a decision-level metric that can measure the intersection of correct predictions between the base model and its quantized variant. We use this metric across multiple models and quantization bit levels (8-bit to 2-bit), and we find that the base and quantized variants usually have a shift in behavior even when accuracy and perplexity are preserved. In order to explain this effect, we study the effect of quantization on the structure of the attention weights using statistical and distributional measures. The results reveal a breakpoint at low bit widths and show that query and key projections are more sensitive to quantization than the value and output projections. These results prove the illusion of equivalency between the base and quantized models and inspire behavioral evaluation beyond perplexity and accuracy for quantization methods.
Abstract:Post-Training Quantization has become widely used to compress large language models to make them deployable on resource-constrained devices. However, the evaluation of quantization methods mainly uses accuracy and perplexity, which cannot capture the behavioral changes in the quantized variants. In this work, we propose Correctness Agreement, a decision-level metric that can measure the intersection of correct predictions between the base model and its quantized variant. We use this metric across multiple models and quantization bit levels (8-bit to 2-bit), and we find that the base and quantized variants usually have a shift in behavior even when accuracy and perplexity are preserved. In order to explain this effect, we study the effect of quantization on the structure of the attention weights using statistical and distributional measures. The results reveal a breakpoint at low bit widths and show that query and key projections are more sensitive to quantization than the value and output projections. These results prove the illusion of equivalency between the base and quantized models and inspire behavioral evaluation beyond perplexity and accuracy for quantization methods.
Artificial Intelligence (cs.AI)
arXiv:2607.08734 [cs.AI]
(or arXiv:2607.08734v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2607.08734 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 268 words
Related coverage
Symmetry-enriched topological order in tensor networks: Defects, gauging and anyon condensation
Quantum Journal · Sep 1, 2026
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
METR · Jul 21, 2026
Quantum Measurement, Information, and Time
scienceblog.com · Sep 23, 2026
Scaling Collider Event Generation with Residual-Quantized Tokens
arXiv · Oct 2, 2026