aiqbench analysis
This research introduces RECAL, a calibration technique designed to mitigate capability degradation in pruned reasoning models before on-policy distillation recovery. By leveraging forward KL divergence to preserve teacher-supported predictions, the method significantly boosts mathematical and code generation performance across various architectures. The work addresses a critical bottleneck in deploying efficient large language models by ensuring that structural compression does not permanently compromise complex reasoning tasks.
ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery
arXiv:2610.11332v1 Announce Type: new Abstract: Structured pruning reduces the deployment cost of reasoning language models, but the resulting capability degradation can hinder subsequent on-policy distillation (OPD) recovery. Because OPD relies on student-generated trajectories, pruning damage that persists after offline distillation can limit its effectiveness. We propose RECAL, Recovery-Aware Calibration, a simple plug-and-play approach that improves OPD recovery by adjusting calibration before pruning. RECAL uses forward KL between an unpruned teacher and a pruned probe to identify teacher-supported predictions disrupted by pruning, then reweights calibration statistics to guide existing pruning criteria toward preserving these predictions. Across multiple models and pruning methods, RECAL consistently improves mathematical reasoning after OPD, achieving gains of up to 16.7 percentage points on AIME, alongside improvements in most code-generation comparisons. Further analysis shows that RECAL reduces residual damage at heavily affected tokens and establishes performance advantages that persist through recovery. These results demonstrate the value of recovery-aware calibration for improving on-policy distillation recovery of pruned reasoning models.
Abstract:Structured pruning reduces the deployment cost of reasoning language models, but the resulting capability degradation can hinder subsequent on-policy distillation (OPD) recovery. Because OPD relies on student-generated trajectories, pruning damage that persists after offline distillation can limit its effectiveness. We propose RECAL, Recovery-Aware Calibration, a simple plug-and-play approach that improves OPD recovery by adjusting calibration before pruning. RECAL uses forward KL between an unpruned teacher and a pruned probe to identify teacher-supported predictions disrupted by pruning, then reweights calibration statistics to guide existing pruning criteria toward preserving these predictions. Across multiple models and pruning methods, RECAL consistently improves mathematical reasoning after OPD, achieving gains of up to 16.7 percentage points on AIME, alongside improvements in most code-generation comparisons. Further analysis shows that RECAL reduces residual damage at heavily affected tokens and establishes performance advantages that persist through recovery. These results demonstrate the value of recovery-aware calibration for improving on-policy distillation recovery of pruned reasoning models.
Computation and Language (cs.CL)
arXiv:2610.11332 [cs.CL]
(or arXiv:2610.11332v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.11332 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 250 words
Related coverage
TAP: Efficient Long-Horizon Agent Pruning via Trajectory-Anchored Recovery
arXiv · Oct 8, 2026
ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
arXiv · Aug 18, 2026
LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry
arXiv · Sep 11, 2026
LiveLaw Supreme Court Half-Yearly Digest: January - June 2026 [Part 2]
livelaw.in · Sep 13, 2026