aiqbench analysis
PADD offers a practical solution for scaling model capacity within fixed computation budgets by enabling dense teachers to effectively train Mixture-of-Experts students without explicit routing. The framework achieves substantial performance gains on mathematical reasoning benchmarks while maintaining stable routing behavior and inference costs comparable to baselines. This approach is particularly relevant for researchers seeking to optimize knowledge distillation pipelines and improve the efficiency of large language models.
PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning
arXiv:2606.10369v1 Announce Type: cross Abstract: As large language models (LLMs) continue to scale, it becomes increasingly challenging to grow model capacity under fixed computation budgets. We propose Path-Aligned Decompression Distillation (PADD), a framework for distilling knowledge from dense teachers without explicit routing into mixture-of-experts (MoE) students while learning high-quality routing policies. PADD organizes knowledge distillation into four stages in two phases: an initialization phase (Stage I) that builds diverse functionality in the student's experts through teacher neuron clustering and student-expert warmup, and a training phase (Stages II--IV) that integrates online adaptive distillation, path-refined policy optimization, and reward-augmented load balancing in a single training pipeline. Experiments on mathematical reasoning benchmarks demonstrate that PADD yields substantial gains over strong baselines at the same inference cost and that the MoE student can match or surpass its dense teacher. They also demonstrate effective teacher-to-student knowledge distillation and stable routing behavior.
Abstract:As large language models (LLMs) continue to scale, it becomes increasingly challenging to grow model capacity under fixed computation budgets. We propose Path-Aligned Decompression Distillation (PADD), a framework for distilling knowledge from dense teachers without explicit routing into mixture-of-experts (MoE) students while learning high-quality routing policies. PADD organizes knowledge distillation into four stages in two phases: an initialization phase (Stage I) that builds diverse functionality in the student's experts through teacher neuron clustering and student-expert warmup, and a training phase (Stages II--IV) that integrates online adaptive distillation, path-refined policy optimization, and reward-augmented load balancing in a single training pipeline. Experiments on mathematical reasoning benchmarks demonstrate that PADD yields substantial gains over strong baselines at the same inference cost and that the MoE student can match or surpass its dense teacher. They also demonstrate effective teacher-to-student knowledge distillation and stable routing behavior.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2606.10369 [cs.CL]
(or arXiv:2606.10369v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.10369 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 239 words
Related coverage
Reward-Aligned Reweighting for On-Policy Distillation
arXiv · Sep 30, 2026
The generative AI customization spectrum: From prompt engineering to custom models on AWS
AWS · Sep 14, 2026
Exploring self-distilled reasoning for supervised fine-tuning with Amazon Nova
AWS · Jul 21, 2026
Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations
arXiv · Aug 3, 2026