analiza aiqbench
This research challenges the industry standard of selecting the highest-performing teacher model for distillation, demonstrating that a "stronger" teacher does not always yield better student outcomes. The authors introduce Student-Centric Answer Sampling (SCAS), which leverages token-wise gradient decomposition to identify supervision that minimizes learning cost for specific student models. By prioritizing matched supervision over raw teacher capability, this approach offers a more efficient path to improving student performance across diverse tasks.
The Strongest Teacher Is Not Always the Best Teacher: Student-Centric Answer Selection
arXiv:2605.26872v2 Announce Type: replace Abstract: LLM training increasingly relies on teacher-generated supervision, from synthetic responses to reasoning traces and tool-use demonstrations. Current practice often chooses the highest-performing teacher to generate student training data, implicitly treating teacher test performance as a proxy for teaching quality. We show that this assumption can fail: even when multiple teachers provide correct answers to the same question, the answer from the strongest teacher is not necessarily the best supervision for a given student. To address this gap, we propose Student-Centric Answer Sampling (SCAS), a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost. Motivated by a token-wise gradient decomposition, we derive an efficient forward-only proxy for this cost and use it to guide answer selection during training. Experiments across 30 teacher models, 6 student base models, and 6 tasks show that SCAS consistently improves student performance, suggesting that effective distillation should prioritize supervision matched to the current student rather than teacher strength alone.
Abstract:LLM training increasingly relies on teacher-generated supervision, from synthetic responses to reasoning traces and tool-use demonstrations. Current practice often chooses the highest-performing teacher to generate student training data, implicitly treating teacher test performance as a proxy for teaching quality. We show that this assumption can fail: even when multiple teachers provide correct answers to the same question, the answer from the strongest teacher is not necessarily the best supervision for a given student. To address this gap, we propose Student-Centric Answer Sampling (SCAS), a framework that selects from verified teacher-generated answers according to their estimated student-centric learning cost. Motivated by a token-wise gradient decomposition, we derive an efficient forward-only proxy for this cost and use it to guide answer selection during training. Experiments across 30 teacher models, 6 student base models, and 6 tasks show that SCAS consistently improves student performance, suggesting that effective distillation should prioritize supervision matched to the current student rather than teacher strength alone.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2605.26872 [cs.LG]
(or arXiv:2605.26872v2 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 176 cuvinte
Subiecte conexe
When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection
arXiv · 1 sept. 2026
I Did This Without Your Knowledge
LessWrong · 17 oct. 2026
In open RLVR, “improvement” depends on the instrument — a small GRPO testbed separating what training optimizes, measures, and teaches
LessWrong · 15 iun. 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · 19 mai 2026