aiqbench analysis
LoRi introduces a low-rank distillation framework that effectively transfers implicit reasoning capabilities to student models by aligning hidden-state trajectories in a shared tensor subspace. This approach consistently bridges the performance gap between implicit and explicit chain-of-thought methods, particularly on complex multi-step mathematical tasks across various model families. The technique offers a promising path for compacting latent reasoning processes without sacrificing accuracy on challenging benchmarks.
LoRi: Low-Rank Distillation for Implicit Reasoning
arXiv:2606.05315v1 Announce Type: cross Abstract: Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting. We empirically find that hidden-state reasoning trajectories exhibit low-rank structure. Motivated by this observation, we propose a low-rank distillation framework that transfers reasoning by aligning teacher and student trajectories in a shared low-rank tensor subspace using first- and second-order statistics. The resulting formulation captures the global structure of reasoning while supporting a compact latent reasoning process. We evaluate the method across multiple model families, including LLaMA and Qwen, at different scales on mathematical reasoning benchmarks. Our approach consistently improves performance, especially on challenging multi-step tasks, approaching explicit CoT accuracy and outperforming prior iCoT distillation methods.
Abstract:Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting. We empirically find that hidden-state reasoning trajectories exhibit low-rank structure. Motivated by this observation, we propose a low-rank distillation framework that transfers reasoning by aligning teacher and student trajectories in a shared low-rank tensor subspace using first- and second-order statistics. The resulting formulation captures the global structure of reasoning while supporting a compact latent reasoning process. We evaluate the method across multiple model families, including LLaMA and Qwen, at different scales on mathematical reasoning benchmarks. Our approach consistently improves performance, especially on challenging multi-step tasks, approaching explicit CoT accuracy and outperforming prior iCoT distillation methods.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.05315 [cs.CL]
(or arXiv:2606.05315v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.05315 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 211 words
Related coverage
LoRi: Low-Rank Distillation for Implicit Reasoning
arXiv · Oct 10, 2026
Some Interesting Papers on RLVR
LessWrong · Jun 9, 2026
Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
arXiv · Aug 21, 2026
Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning
arXiv · Sep 15, 2026