aiqbench analysis
This research introduces LoRi, a novel distillation framework that leverages the low-rank structure of hidden-state reasoning trajectories to improve implicit chain-of-thought performance in large language models. By aligning teacher and student models within a shared tensor subspace, the method successfully bridges the accuracy gap between implicit internalized reasoning and explicit prompting on complex mathematical tasks. The findings offer a significant advancement for efficient model compression and reasoning capability transfer across various architectures like LLaMA and Qwen.
LoRi: Low-Rank Distillation for Implicit Reasoning
arXiv:2606.05315v2 Announce Type: replace-cross Abstract: Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting. We empirically find that hidden-state reasoning trajectories exhibit low-rank structure. Motivated by this observation, we propose a low-rank distillation framework that transfers reasoning by aligning teacher and student trajectories in a shared low-rank tensor subspace using first- and second-order statistics. The resulting formulation captures the global structure of reasoning while supporting a compact latent reasoning process. We evaluate the method across multiple model families, including LLaMA and Qwen, at different scales on mathematical reasoning benchmarks. Our approach consistently improves performance, especially on challenging multi-step tasks, approaching explicit CoT accuracy and outperforming prior iCoT distillation methods.
Abstract:Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting. We empirically find that hidden-state reasoning trajectories exhibit low-rank structure. Motivated by this observation, we propose a low-rank distillation framework that transfers reasoning by aligning teacher and student trajectories in a shared low-rank tensor subspace using first- and second-order statistics. The resulting formulation captures the global structure of reasoning while supporting a compact latent reasoning process. We evaluate the method across multiple model families, including LLaMA and Qwen, at different scales on mathematical reasoning benchmarks. Our approach consistently improves performance, especially on challenging multi-step tasks, approaching explicit CoT accuracy and outperforming prior iCoT distillation methods.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.05315 [cs.CL]
(or arXiv:2606.05315v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.05315 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 209 words
Related coverage
LoRi: Low-Rank Distillation for Implicit Reasoning
arXiv · Jun 6, 2026
Some Interesting Papers on RLVR
LessWrong · Jun 9, 2026
Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
arXiv · Aug 21, 2026
Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning
arXiv · Sep 15, 2026