analiza aiqbench
This research introduces EBM-RL, a reinforcement learning framework that decouples visual perception, reasoning, and dialogue generation to enhance immersive video role-playing. By mimicking the human See-Think-Speak process with specific reward signals for scene alignment and faithfulness, the method significantly outperforms text-only baselines in capturing atmospheric tension and character authenticity. The work also contributes an open-source dataset for video-grounded dialogue, advancing the state of multimodal AI agents in interactive narratives.
Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playing
arXiv:2605.04733v3 Announce Type: replace Abstract: Text-based role-playing models can imitate character styles, but often fail to capture scene atmosphere and evolving tension, which are crucial for immersive applications such as VR games and interactive narratives. We study video-grounded role-playing dialogue and introduce EBM-RL (Eye--Brain--Mouth Reinforcement Learning), a decoupled GRPO-based framework that separates observation ( ), reasoning (), and utterance generation (). This design mimics the human See-Think-Speak process, enabling the model to ground dialogue in visual perception before reasoning and response generation. To optimize this See-Think-Speak process, EBM-RL integrates complementary rewards for scene--text alignment, perceptual--cognitive utility, answer faithfulness, and format consistency. Extensive experiments show that EBM-RL substantially outperforms text-only role-playing baselines and larger-scale vision-language models on our immersive role-playing benchmark, improving both visual-atmosphere consistency and character authenticity. Moreover, EBM-RL demonstrates strong zero-shot transfer to out-of-domain VideoQA benchmarks without additional fine-tuning. We also release an open-source dataset for video-grounded role-playing dialogue.
Abstract:Text-based role-playing models can imitate character styles, but often fail to capture scene atmosphere and evolving tension, which are crucial for immersive applications such as VR games and interactive narratives. We study video-grounded role-playing dialogue and introduce EBM-RL (Eye--Brain--Mouth Reinforcement Learning), a decoupled GRPO-based framework that separates observation (<perception>), reasoning (<think>), and utterance generation (<answer>). This design mimics the human See-Think-Speak process, enabling the model to ground dialogue in visual perception before reasoning and response generation. To optimize this See-Think-Speak process, EBM-RL integrates complementary rewards for scene--text alignment, perceptual--cognitive utility, answer faithfulness, and format consistency. Extensive experiments show that EBM-RL substantially outperforms text-only role-playing baselines and larger-scale vision-language models on our immersive role-playing benchmark, improving both visual-atmosphere consistency and character authenticity. Moreover, EBM-RL demonstrates strong zero-shot transfer to out-of-domain VideoQA benchmarks without additional fine-tuning. We also release an open-source dataset for video-grounded role-playing dialogue.
Artificial Intelligence (cs.AI)
arXiv:2605.04733 [cs.AI]
(or arXiv:2605.04733v3 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.04733 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 236 cuvinte
Subiecte conexe
Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playing
arXiv · 6 iun. 2026
Jev: System One models for Prod, not God — with Diogo Almeida, CEO, TypeSafe AI
Latent Space · 21 sept. 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · 12 iun. 2026
Microsoft’s AI chief says superintelligence is near, but won’t take your job
The Verge · 8 iun. 2026