Emotion Recognition in Sign Language Conversation

Abstract:Emotion Recognition in Conversation is a core component of affective computing, while current sign language emotion datasets primarily focus on isolated sentences and lack conversational context. Models trained exclusively on these isolated utterances demonstrate degraded performance in real world scenarios because they cannot utilize historical dialogue flow. To address this structural limitation, we introduce the ERC task to sign language video analysis and propose the eJSL Dialog dataset. Constructed using the scripts from the STUDIES corpus, the dataset contains 1,920 video samples organized into 480 unique dialogues. We conduct systematic benchmarking on this dataset using models ranging from isolated visual networks to multimodal conversational architectures. The results suggest the feasibility of extending conversational ERC frameworks to sign-language dialogue under the current benchmark setting, while also revealing limitations in existing visual representations for capturing sign-specific affective cues, motivating future work on sign-specific visual modeling and larger sign-language conversational training resources.

Computation and Language (cs.CL)

arXiv:2605.23328 [cs.CL]

(or arXiv:2605.23328v3 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2605.23328 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.