Speech Signals Complement LLMs for Predicting Interpersonal Attraction in Speed Dating

Abstract:Large language models (LLMs) can predict interpersonal attraction from conversation transcripts, but it remains unclear what a speech predictor can add beyond transcript-only LLM prediction. Using Japanese speed-dating conversations, we combine predictions from a transcript-only LLM and a supervised speech predictor to estimate participants' reported liking of their partners. We show that speech can complement transcript-only LLM prediction, but that this complementarity is conditional rather than universal. Combining the two predictions significantly improves pairwise ranking accuracy over the transcript-only LLM alone in all evaluated conditions. By contrast, gains in per-participant Pearson $r$ vary across conversation rounds and rating directions, with none significant after correction. Retrospectively, these $r$ gains are concentrated among participants for whom the speech predictor is more accurate. Speech can therefore retain predictive value even when an LLM predicts attraction from transcripts. The relevant question is not simply whether speech helps, but where its complementarity emerges.

Accepted at the 28th ACM International Conference on Multimodal Interaction (ICMI 2026). 16 pages total, including a 7-page supplementary appendix

Computation and Language (cs.CL)

arXiv:2607.23037 [cs.CL]

(or arXiv:2607.23037v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2607.23037 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

https://doi.org/10.1145/3776574.3831151 Focus to learn more DOI(s) linking to related resources

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.