aiqbench analysis
This preprint studies how to make pruned large reasoning models produce shorter chains of thought. It proposes Contrastive Set-Difference, which prunes neurons that activate more on long correct solutions than on short ones, and extends preference-based length compression to problems a pruned model always or never solves. On DeepSeek-R1-Distill-Qwen-7B at 20% pruning, the authors report 8-22% fewer thinking tokens across four math benchmarks with accuracy within 0.5 points, but this is a preliminary version evaluated on math only, so the gains need broader validation.
Improving the Chain-of-Thought Compressibility of Pruned Reasoning Models via Long--Short Contrastive Pruning
arXiv:2610.34336v1 Announce Type: cross Abstract: Large reasoning models are costly to serve because they are large and because they generate long chains of thought (CoT). Weight pruning and CoT compression address these two costs separately, but they interfere in that pruning adds redundant reasoning on problems the model still solves and incomplete reasoning on those it no longer solves, and on harder problems makes the model less compressible by subsequent length-preference optimization. We trace this to a blind spot on each side: pruning criteria do not distinguish necessary from redundant reasoning, and preference-based CoT compression receives no training signal on problems a pruned model always or never solves. We propose Contrastive Set-Difference (CSD), which identifies neurons enriched in redundant reasoning by contrasting their activations on long and short correct solutions of the same problem and prunes them preferentially, and we extend preference pairs to the problems a pruned model always or never solves. Pruning 20\% of DeepSeek-R1-Distill-Qwen-7B, the combination uses 8--22\% fewer thinking tokens on four math benchmarks than pruning with reasoning-aware calibration (RAC) followed by the same compression, with accuracy within 0.5 points.
Abstract:Large reasoning models are costly to serve because they are large and because they generate long chains of thought (CoT). Weight pruning and CoT compression address these two costs separately, but they interfere in that pruning adds redundant reasoning on problems the model still solves and incomplete reasoning on those it no longer solves, and on harder problems makes the model less compressible by subsequent length-preference optimization. We trace this to a blind spot on each side: pruning criteria do not distinguish necessary from redundant reasoning, and preference-based CoT compression receives no training signal on problems a pruned model always or never solves. We propose Contrastive Set-Difference (CSD), which identifies neurons enriched in redundant reasoning by contrasting their activations on long and short correct solutions of the same problem and prunes them preferentially, and we extend preference pairs to the problems a pruned model always or never solves. Pruning 20\% of DeepSeek-R1-Distill-Qwen-7B, the combination uses 8--22\% fewer thinking tokens on four math benchmarks than pruning with reasoning-aware calibration (RAC) followed by the same compression, with accuracy within 0.5 points.
11 pages. Preliminary version; further experiments and revisions are planned
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2610.34336 [cs.CL]
(or arXiv:2610.34336v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.34336 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 287 words