Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?

Abstract:Current full-duplex (FD) spoken dialogue systems can produce fluid interactions, yet it remains unclear whether they can adapt their turn-taking behavior when explicitly instructed. This is critical for real-world deployment, where conversational policies vary across applications (e.g., proactive tutoring vs. passive counseling). We introduce Instruct-FD, an instruction-conditioned benchmark for evaluating controllable turn management in FD systems. To enable this, we develop a human-validated, scalable synthetic pipeline that generates instruction-conditioned conversations, along with a deployment-agnostic multi-turn evaluation protocol and an LLM-based judge. Benchmarking six state-of-the-art full-duplex systems reveals a substantial gap in instruction-following turn management: the best model achieves only 64.4% adherence. Performance is highly uneven across behaviors and scenarios, with proactive behaviors such as model backchanneling and interruption remaining particularly challenging. These findings establish instruction-following turn management as a crucial direction for building adaptable and deployable full-duplex dialogue systems.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

arXiv:2607.20460 [cs.CL]

(or arXiv:2607.20460v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2607.20460 arXiv-issued DOI via DataCite (pending registration)