aiqbench analysis
This research addresses the training efficiency bottleneck in speech foundation model (SFM) distillation by introducing interleaved stacking, a technique that preserves layer-specific knowledge to prevent performance degradation during depth progression. By accelerating the student model training process without sacrificing accuracy, the method offers a practical pathway for faster deployment of efficient speech models in low-resource environments. The approach is validated on the SUPERB benchmark, demonstrating its viability for optimizing inference latency in audio processing pipelines.
Fast Speech Foundation Model Distillation Using Interleaved Stacking
arXiv:2606.11766v1 Announce Type: cross Abstract: Distilling a large speech foundation model (SFM) into an efficient student model has been successfully applied to low-resource environments. Although distillation reduces inference latency, it requires an additional student model training. However, the training efficiency of SFM distillation remains underexplored. In this work, we explore training acceleration of SFM distillation to speed up model deployment. We examine the potential of stacking, in which the model depth is progressively increased through training until the target model depth is reached. While existing stacking methods improve training speed, they suffer from performance degradation. To handle this limitation, we propose interleaved stacking, a novel stacking method that consistently preserves layer position throughout the stacking process. This property is particularly critical in SFMs, in which each layer encodes distinct layer-specific knowledge. We validate the effectiveness of the proposed method on SUPERB.
Abstract:Distilling a large speech foundation model (SFM) into an efficient student model has been successfully applied to low-resource environments. Although distillation reduces inference latency, it requires an additional student model training. However, the training efficiency of SFM distillation remains underexplored. In this work, we explore training acceleration of SFM distillation to speed up model deployment. We examine the potential of stacking, in which the model depth is progressively increased through training until the target model depth is reached. While existing stacking methods improve training speed, they suffer from performance degradation. To handle this limitation, we propose interleaved stacking, a novel stacking method that consistently preserves layer position throughout the stacking process. This property is particularly critical in SFMs, in which each layer encodes distinct layer-specific knowledge. We validate the effectiveness of the proposed method on SUPERB.
Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Sound (cs.SD)
arXiv:2606.11766 [eess.AS]
(or arXiv:2606.11766v1 [eess.AS] for this version)
https://doi.org/10.48550/arXiv.2606.11766 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 241 words
Related coverage
Fast Speech Foundation Model Distillation Using Interleaved Stacking
arXiv · Jun 17, 2026
How Much Do Teacher Bridges Matter in Interleaved On-Policy Distillation?
arXiv · Oct 30, 2026
Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?
arXiv · Jul 21, 2026
Pistis Technical Report
arXiv · Sep 25, 2026