Do Visual Features Improve Other-Initiated Repair Detection? A Dyadic Multimodal Approach

Title:Do Visual Features Improve Other-Initiated Repair Detection? A Dyadic Multimodal Approach

Abstract:Other-initiated Self-repair, or in short Other-initiated Repair (OIR), is an essential mechanism in conversational interaction, whereby a recipient signals a problem in speaking, hearing, or understanding, prompting the previous speaker to resolve it. In the case of conversational agents, it is essential to accurately identify these repair initiation strategies to address communication breakdowns efficiently. While conversational analysis studies have shown that OIR initiation is accompanied by both verbal and non-verbal signals such as gaze shifts, facial expressions, body postures, and hand gestures, existing computational approaches rely mainly on text and audio. This paper introduces a novel multimodal model for OIR detection and classification, incorporating a set of visual features drawn from conversation analysis. We evaluate our approach on two corpora with distinct languages and interaction settings. Results demonstrate that visual information consistently improves performance over text and audio baselines, and provide insights into cross-modal feature contributions across two corpora.

Artificial Intelligence (cs.AI)

arXiv:2607.23845 [cs.AI]

(or arXiv:2607.23845v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2607.23845 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

https://doi.org/10.1145/3776574.3831200 Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.