aiqbench analysis
This research challenges the industry standard of selecting the highest-performing teacher model for distillation, demonstrating that student-specific learning costs often outweigh raw teacher capability. The proposed Student-Centric Answer Sampling (SCAS) framework utilizes gradient decomposition to efficiently identify optimal supervision signals tailored to individual student models. By prioritizing matched supervision over sheer teacher strength, this approach offers a more effective strategy for training large language models across diverse tasks and architectures.
When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection
arXiv:2605.26872v4 Announce Type: replace-cross Abstract: LLM training increasingly relies on teacher-generated supervision, from synthetic responses to reasoning traces and tool-use demonstrations. Current practice often chooses the highest-performing teacher to generate student training data, implicitly treating teacher test performance as a proxy for teaching quality. We show that this assumption can fail: even when multiple teachers provide correct answers to the same question, the answer from the strongest teacher is not necessarily the best supervision for a given student. To address this gap, we propose Student-Centric Answer Sampling (SCAS), a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost. Motivated by a token-wise gradient decomposition, we derive an efficient forward-only proxy for this cost and use it to guide answer selection during training. Experiments across 30 teacher models, 6 student base models, and 6 tasks show that SCAS consistently improves student performance, suggesting that effective distillation should prioritize supervision matched to the current student rather than teacher strength alone.
Abstract:LLM training increasingly relies on teacher-generated supervision, from synthetic responses to reasoning traces and tool-use demonstrations. Current practice often chooses the highest-performing teacher to generate student training data, implicitly treating teacher test performance as a proxy for teaching quality. We show that this assumption can fail: even when multiple teachers provide correct answers to the same question, the answer from the strongest teacher is not necessarily the best supervision for a given student. To address this gap, we propose Student-Centric Answer Sampling (SCAS), a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost. Motivated by a token-wise gradient decomposition, we derive an efficient forward-only proxy for this cost and use it to guide answer selection during training. Experiments across 30 teacher models, 6 student base models, and 6 tasks show that SCAS consistently improves student performance, suggesting that effective distillation should prioritize supervision matched to the current student rather than teacher strength alone.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2605.26872 [cs.LG]
(or arXiv:2605.26872v4 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.26872 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 257 words
Related coverage
The Strongest Teacher Is Not Always the Best Teacher: Student-Centric Answer Selection
arXiv · Jun 9, 2026
Leaky Students: Membership Inference against On-Policy Distillation
arXiv · Sep 30, 2026
OPOD: On-Policy Omni Distillation
arXiv · Jul 24, 2026
[Paper] Distillation for Incrimination and Distillation for Capabilities
LessWrong · Oct 9, 2026