aiqbench analysis
This paper is squarely about machine learning: it studies knowledge distillation during LLM pretraining and finds that teacher strength matters less than assumed, since small, undertrained teachers can help larger students when loss terms are mixed properly. The more useful takeaway for practitioners is that scaling up the teacher can saturate or reverse gains, so teacher selection and loss weighting deserve tuning rather than default assumptions. These are controlled experiments on one setup, so the findings should be validated on other architectures and data mixes before being treated as general rules.
Strong Teacher Not Needed? On Distillation in LLM Pretraining
arXiv:2605.23857v2 Announce Type: replace Abstract: Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield better students. In this work, we examine this assumption about distillation in large language model pretraining. By varying architecture sizes and training token budgets, we create strong-to-weak, same-level, and weak-to-strong teacher-student relationships, and study distillation's effectiveness under each. We find that the teacher need not be strong: with proper mixing of the language modeling and knowledge distillation losses, even small and undertrained teachers improve larger students. At the same time, a stronger teacher is not always better: pushing the teacher further, through more parameters or more training tokens, can saturate or even reverse the distillation gains. We further observe that distillation improves generalization (out-of-distribution and downstream performance) more readily than in-domain fitting. Together, these results challenge the common belief that distillation pretraining always requires a strong teacher.
Title:Strong Teacher Not Needed? On Distillation in LLM Pretraining
Abstract:Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield better students. In this work, we examine this assumption about distillation in large language model pretraining. By varying architecture sizes and training token budgets, we create strong-to-weak, same-level, and weak-to-strong teacher-student relationships, and study distillation's effectiveness under each. We find that the teacher need not be strong: with proper mixing of the language modeling and knowledge distillation losses, even small and undertrained teachers improve larger students. At the same time, a stronger teacher is not always better: pushing the teacher further, through more parameters or more training tokens, can saturate or even reverse the distillation gains. We further observe that distillation improves generalization (out-of-distribution and downstream performance) more readily than in-domain fitting. Together, these results challenge the common belief that distillation pretraining always requires a strong teacher.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2605.23857 [cs.LG]
(or arXiv:2605.23857v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.23857 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 243 words
Related coverage
DISTAL: Distillation and Self-Supervised Pretraining for Structure-Agnostic Materials Property Prediction
arXiv · Sep 2, 2026
PA1-OPD: One-Step Prefix-Augmented On-Policy Distillation for Language Model Pretraining
arXiv · Oct 15, 2026
RAPiD: Reward-Guided Consistency Distillation of Diffusion Planners for Real-Time Autonomous Driving
arXiv · Aug 3, 2026
Self-Supervised Representation-Guided Generative Dataset Distillation
arXiv · Aug 5, 2026