Does Listening Matter? Backchanneling and Nodding in AI Clone

Title:Does Listening Matter? Backchanneling and Nodding in AI Clone

Abstract:AI clones that imitate a specific person typically reproduce what the person says and how they sound, but not how they listen. We investigate whether adding multimodal listening behaviors gives such a clone more presence and authenticity. We integrated verbal backchannels and head nodding, driven by real-time prediction models, into an AI clone equipped with voice cloning and LLM-based responses. In a within-subjects study (N=35), adding these behaviors significantly improved the perceived attentiveness of the avatar, the sense of talking with the real person, and the feeling of co-presence. These results indicate that AI clone fidelity should extend beyond voice and response content to include interactive listening behavior.

Human-Computer Interaction (cs.HC); Computation and Language (cs.CL); Sound (cs.SD)

arXiv:2608.19527 [cs.HC]

(or arXiv:2608.19527v1 [cs.HC] for this version)

https://doi.org/10.48550/arXiv.2608.19527 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.