aiqbench analysis
This research introduces attention-based policies that enable pre-trained speech-to-text models to operate effectively in streaming scenarios without retraining, significantly reducing translation latency while maintaining high accuracy. By leveraging cross-attention mechanisms to align audio frames with text tokens, the proposed methods offer a practical solution for real-time simultaneous interpretation tasks. The findings demonstrate substantial performance gains across multiple language pairs, highlighting the potential for deploying existing large models in low-latency applications.
Attention-Based Adaptive Policies for Simultaneous Speech-to-Text Translation
arXiv:2609.30839v1 Announce Type: new Abstract: Simultaneous speech-to-text translation (Simul-S2TT) consists of generating partial translations while the incoming audio frames are processed by the system. However, the streaming nature of this setup creates the challenge of deciding the best moment to perform an accurate translation while minimizing the delay. To address this challenge, we utilize the cross-attention mechanism of the encoder-decoder architecture to find the right alignment between the input speech frames and the target text tokens. In this paper, we propose the Recent Frame Attention Policy (RFAP) and the Dual-Condition Attention Policy (DCAP) that allow offline trained speech-to-text translation models to be used in streaming scenarios without requiring additional training. Results on three different language translation pairs over the CVSS-C corpus show that the RFAP is able to surpass other policies with gains of up to 4.0 BLEU while reducing the translation delay by almost 1 second. Moreover, the DCAP is able to preserve a high translation quality when the latency is very low.
Abstract:Simultaneous speech-to-text translation (Simul-S2TT) consists of generating partial translations while the incoming audio frames are processed by the system. However, the streaming nature of this setup creates the challenge of deciding the best moment to perform an accurate translation while minimizing the delay. To address this challenge, we utilize the cross-attention mechanism of the encoder-decoder architecture to find the right alignment between the input speech frames and the target text tokens. In this paper, we propose the Recent Frame Attention Policy (RFAP) and the Dual-Condition Attention Policy (DCAP) that allow offline trained speech-to-text translation models to be used in streaming scenarios without requiring additional training. Results on three different language translation pairs over the CVSS-C corpus show that the RFAP is able to surpass other policies with gains of up to 4.0 BLEU while reducing the translation delay by almost 1 second. Moreover, the DCAP is able to preserve a high translation quality when the latency is very low.
Machine Learning (cs.LG)
arXiv:2609.30839 [cs.LG]
(or arXiv:2609.30839v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.30839 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 253 words
Related coverage
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · Sep 17, 2026
Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture
arXiv · Sep 22, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026
A Red Line and Oversight Framework for Government AI Contracts
AI Alignment Forum · Jul 18, 2026