aiqbench analysis
This research addresses the training efficiency bottleneck in distilling large speech foundation models by introducing interleaved stacking, a technique that preserves layer-specific knowledge during progressive depth expansion. By mitigating performance degradation common in existing stacking methods, this approach accelerates model deployment for low-resource environments without sacrificing accuracy on benchmarks like SUPERB. The work represents a significant methodological advance specifically within the domain of machine learning and speech processing architectures.
Fast Speech Foundation Model Distillation Using Interleaved Stacking
arXiv:2606.11766v2 Announce Type: replace-cross Abstract: Distilling a large speech foundation model (SFM) into an efficient student model has been successfully applied to low-resource environments. Although distillation reduces inference latency, it requires an additional student model training. However, the training efficiency of SFM distillation remains underexplored. In this work, we explore training acceleration of SFM distillation to speed up model deployment. We examine the potential of stacking, in which the model depth is progressively increased through training until the target model depth is reached. While existing stacking methods improve training speed, they suffer from performance degradation. To handle this limitation, we propose interleaved stacking, a novel stacking method that consistently preserves layer position throughout the stacking process. This property is particularly critical in SFMs, in which each layer encodes distinct layer-specific knowledge. We validate the effectiveness of the proposed method on SUPERB.
Abstract:Distilling a large speech foundation model (SFM) into an efficient student model has been successfully applied to low-resource environments. Although distillation reduces inference latency, it requires an additional student model training. However, the training efficiency of SFM distillation remains underexplored. In this work, we explore training acceleration of SFM distillation to speed up model deployment. We examine the potential of stacking, in which the model depth is progressively increased through training until the target model depth is reached. While existing stacking methods improve training speed, they suffer from performance degradation. To handle this limitation, we propose interleaved stacking, a novel stacking method that consistently preserves layer position throughout the stacking process. This property is particularly critical in SFMs, in which each layer encodes distinct layer-specific knowledge. We validate the effectiveness of the proposed method on SUPERB.
Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Sound (cs.SD)
arXiv:2606.11766 [eess.AS]
(or arXiv:2606.11766v2 [eess.AS] for this version)
https://doi.org/10.48550/arXiv.2606.11766 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 239 words
Related coverage
Fast Speech Foundation Model Distillation Using Interleaved Stacking
arXiv · Jun 11, 2026
How Much Do Teacher Bridges Matter in Interleaved On-Policy Distillation?
arXiv · Oct 30, 2026
Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?
arXiv · Jul 21, 2026
Pistis Technical Report
arXiv · Sep 25, 2026