aiqbench analysis
This research introduces On-Policy Delta Distillation (OPD²), a refined post-training technique that leverages probability gaps between teacher and base models to enhance mathematical reasoning across English, Korean, and Japanese. By analyzing Qwen3 experiments, the study demonstrates that OPD² significantly narrows performance disparities in multilingual settings while preventing response drift toward English when using monolingual training data. The findings highlight critical strategies for maintaining linguistic fidelity during LLM fine-tuning without relying on reinforcement learning.
On-Policy Delta Distillation for Multilingual Math Reasoning
arXiv:2608.05802v1 Announce Type: new Abstract: On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, On-Policy Delta Distillation (OPD$^2$), for mathematical reasoning in English, Korean, and Japanese. OPD$^2$ improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD$^2$ consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese, and generally narrows the English-Korean performance gap. We further find that English-only OPD can also increase performance for Korean and Japanese, but often shifts the responses toward English, highlighting the importance of multilingual data to preserving target-language responses.
Abstract:On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, On-Policy Delta Distillation (OPD$^2$), for mathematical reasoning in English, Korean, and Japanese. OPD$^2$ improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD$^2$ consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese, and generally narrows the English-Korean performance gap. We further find that English-only OPD can also increase performance for Korean and Japanese, but often shifts the responses toward English, highlighting the importance of multilingual data to preserving target-language responses.
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2608.05802 [cs.CL]
(or arXiv:2608.05802v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.05802 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 217 words
Related coverage
Enabling Language-specific Reasoning in Multilingual Models with Reinforcement Learning
AMD · Jul 24, 2026
Frontier post-training recipe review with Finbarr Timbers
Interconnects · Jun 16, 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026