JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue Systems

Abstract:Efficient and robust turn-taking detection remains a significant challenge in industrial-grade voice AI agents. Existing systems often rely on acoustic or linguistic cues alone, limiting accuracy and stability, whereas directly endowing large language models with full-duplex capability incurs heavy data and deployment costs that hinder real-time usage. We propose JAL-Turn, a lightweight speech-only turn-taking framework based on a joint acoustic-linguistic modeling paradigm, in which a cross-attention module adaptively fuses pre-trained acoustic and linguistic representations for low-latency hold or shift prediction. By sharing a frozen ASR encoder, JAL-Turn enables turn-taking prediction to run fully in parallel with speech recognition, introducing almost no additional end-to-end latency. We further introduce a scalable pipeline that automatically derives reliable turn-taking labels from large-scale real-world dialogue corpora. Experiments on the public Easy-Turn and Smart-Turn benchmarks and an in-house Japanese customer-service dataset show that JAL-Turn achieves competing results among strong baselines in detection accuracy while maintaining superior real-time performance.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

arXiv:2603.26515 [cs.CL]

(or arXiv:2603.26515v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2603.26515 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.