perspectiva aiqbenchCercetare AI

DuDi addresses the critical degradation of multilingual capabilities in small language models (SLMs) by integrating online sequence-level signals with off-policy and on-policy token-level supervision. The framework's use of a cross-lingual verbalizer effectively refines teacher feedback, enabling robust transferability across Southeast Asian languages where sub-billion scale models typically underperform. This approach offers a scalable pathway to enhance multilingual SLMs without relying on larger teacher models, directly tackling a key bottleneck in efficient multilingual deployment.

Sursă care acoperăarXiv

Ce este aceastaLectura unui model asupra articolului și a acoperirii conexe, scrisă pentru aiqbench. Nu sunt cuvintele publicației.

DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer

arXiv4 iunie 20261 min de cititArticolul și perspectiva sunt în engleză.

arXiv:2606.04694v1 Announce Type: new Abstract: Small language models (SLMs) are efficient and scalable, but their multilingual capabilities degrade severely at sub-billion scales, especially for Southeast Asian (SEA) languages. We introduce DuDi, a dual-signal multilingual distillation framework that combines an online sequence-level signal with off-policy and on-policy token-level signals. DuDi further uses a cross-lingual verbalizer to refine teacher feedback and improve teacher-student transferability in multilingual settings. Experiments on SEA-HELM across multiple model families, scales, and teacher-student settings show that DuDi consistently outperforms competitive distillation baselines. Ablations and analyses confirm that sequence-level optimization, token-level supervision, and cross-lingual verbalization provide complementary and transferable learning signals for multilingual SLMs.

Citește articolul complet pe arXiv

Citește pe arXiv ↗Întreabă despre acest articol