aiqbench analysis
This arXiv paper examines how reinforcement learning changes vision-language model agents internally. It finds that RL-trained gains in grounded visual question answering come mostly from the LLM layers attending more to visual tokens, not from better vision encoders. A fixed attention bias partially recovers those gains without any retraining, which is a useful interpretability result, though it is a single-paper finding across four environments and not yet a general account of RL in multimodal models.
RL Teaches VLMs Where to Attend but Not How to Perceive
arXiv:2610.23983v1 Announce Type: cross Abstract: Although reinforcement learning (RL) effectively trains Vision-Language Model (VLM) agents, the mechanisms behind these capability gains are poorly understood. Across four diverse agent environments, we investigate whether the task-success gains from multi-turn RL are accompanied by improved grounded visual question answering about the environment. To assess this, we design environment-specific grounded VQA tasks that reveal improvements in grounded visual understanding. Representational probing shows that vision-encoder probe accuracy remains relatively stable, while LLM-layer probe accuracy improves. Attention analysis further reveals that LLM layers pay more attention to visual tokens during RL training. To test whether increased attention to visual tokens contributes to VQA gains, we introduce a lightweight plug-and-play intervention using a fixed attention bias. Without updating model parameters or computing gradients, this intervention partially restores VQA gains on the untrained model. These results reveal better utilization of visual information as a key mechanism behind the gains from RL.
Abstract:Although reinforcement learning (RL) effectively trains Vision-Language Model (VLM) agents, the mechanisms behind these capability gains are poorly understood. Across four diverse agent environments, we investigate whether the task-success gains from multi-turn RL are accompanied by improved grounded visual question answering about the environment. To assess this, we design environment-specific grounded VQA tasks that reveal improvements in grounded visual understanding. Representational probing shows that vision-encoder probe accuracy remains relatively stable, while LLM-layer probe accuracy improves. Attention analysis further reveals that LLM layers pay more attention to visual tokens during RL training. To test whether increased attention to visual tokens contributes to VQA gains, we introduce a lightweight plug-and-play intervention using a fixed attention bias. Without updating model parameters or computing gradients, this intervention partially restores VQA gains on the untrained model. These results reveal better utilization of visual information as a key mechanism behind the gains from RL.
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
arXiv:2610.23983 [cs.CV]
(or arXiv:2610.23983v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2610.23983 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 246 words
Related coverage
See, Measure, and Reason: Learning Visually Grounded Reasoning in Pathology
arXiv · Sep 29, 2026
Can you see how I learn? Human observers' inferences about Reinforcement Learning agents' learning processes
arXiv · Aug 24, 2026
Falcon-Emirati: When an LLM Learns the Dialect, the Culture, and the Nuance
Hugging Face · Oct 6, 2026
Using AI Agents to Teach Robots Like Microduck to Walk in Simulation
NVIDIA · Oct 23, 2026