aiqbench analysis
SeRV introduces a novel vector quantization tokenizer that aligns ASL motion representations with semantic text inputs, overcoming previous limitations in generating coherent sign language from linguistic descriptions. By leveraging a hierarchical GPT architecture trained on a reconstructed 3D benchmark, the method achieves state-of-the-art pose accuracy while ensuring structural consistency between generated motions and their textual prompts. This advancement represents a significant step forward in multimodal AI applications for accessibility and human-computer interaction.
SeRV: Semantic-Aligned Residual Vector Quantization for American Sign Language Generation
arXiv:2609.05742v1 Announce Type: cross Abstract: American Sign Language (ASL) generation remains challenging due to limited paired text-ASL motion data and the difficulty of learning motion representations both precise for reconstruction and predictable from linguistic input. Existing methods rely on motion tokenizers optimized for reconstruction, without explicit semantic supervision from paired text. As a result, the learned tokens remain limited in supporting semantically consistent and fine-grained ASL motion generation. To address this limitation, we propose SeRV (Semantic-Aligned Residual Vector Quantization), a semantic-aligned RVQ tokenizer for ASL generation. SeRV learns a semantically structured residual token space by combining sentence-level motion-text alignment with token-level text-conditioned supervision. Building on this tokenizer, a Hierarchical GPT predicts residual motion tokens in a coarse-to-fine manner, generating structurally coherent and semantically aligned 3D ASL motion. We further construct a large-scale reconstructed 3D ASL motion-text benchmark by recovering paired 3D motion from YouTube-ASL videos. Experiments across 375 hours of ASL video show that SeRV achieves state-of-the-art pose accuracy on both How2Sign and YouTube-ASL datasets, while producing semantically consistent 3D ASL motion directly from text.
Abstract:American Sign Language (ASL) generation remains challenging due to limited paired text-ASL motion data and the difficulty of learning motion representations both precise for reconstruction and predictable from linguistic input. Existing methods rely on motion tokenizers optimized for reconstruction, without explicit semantic supervision from paired text. As a result, the learned tokens remain limited in supporting semantically consistent and fine-grained ASL motion generation. To address this limitation, we propose SeRV (Semantic-Aligned Residual Vector Quantization), a semantic-aligned RVQ tokenizer for ASL generation. SeRV learns a semantically structured residual token space by combining sentence-level motion-text alignment with token-level text-conditioned supervision. Building on this tokenizer, a Hierarchical GPT predicts residual motion tokens in a coarse-to-fine manner, generating structurally coherent and semantically aligned 3D ASL motion. We further construct a large-scale reconstructed 3D ASL motion-text benchmark by recovering paired 3D motion from YouTube-ASL videos. Experiments across 375 hours of ASL video show that SeRV achieves state-of-the-art pose accuracy on both How2Sign and YouTube-ASL datasets, while producing semantically consistent 3D ASL motion directly from text.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2609.05742 [cs.CV]
(or arXiv:2609.05742v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2609.05742 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 270 words
Related coverage
Productionizing TurboQuant on AMD GPUs for KV-Cache-Bound LLM Inference
AMD · Jun 11, 2026
X-Tokenizer: A Multimodal Action Tokenizer for Vision-Language-Action Pretraining
arXiv · Jun 16, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
Does Your LLM Trust You?
LessWrong · Aug 4, 2026