aiqbench analysis
RayViT addresses a critical gap in visual imitation learning by integrating explicit camera geometry into Vision Transformer backbones to mitigate policy brittleness under viewpoint changes. This architecture enhances robustness through ray-conditioned representations, demonstrating significant performance gains on both simulated and real-world robotic tasks without requiring heavy computational overhead. The work represents a meaningful advancement in making AI-driven robotics more reliable for dynamic environments where sensor perspectives vary.
RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning
arXiv:2607.29622v1 Announce Type: new Abstract: Visual imitation learning enables robots to acquire visuomotor skills directly from images, yet RGB observations lack explicit geometric cues, making learned policies brittle to camera perturbations. To address this, we propose \textbf{Ray-conditioned Vision Transformer Encoder (RayViT)}, a lightweight architecture that injects camera geometry into pretrained ViT backbones. RayViT represents camera geometry as a Pl\"ucker ray map, patchifies it into ray features, and uses gated cross-attention to produce a ray-conditioned class token. These ray features are added as dense positional embeddings, while the ray class token replaces the original ViT class token to provide a geometry-aware summary representation. We combine this approach with an auxiliary cosine similarity loss to consistently improve the performance and robustness for geometry-aware tokens. Experiments on sim- and real-robot tasks demonstrate that RayViT improves robustness by approximately 13 percentage points under camera perturbations in multi-task RoboCasa benchmark and by 1.78 average completed stages in real-world multi-task success rate compared to baselines.
Authors:Qian Wang, Longrui Chen, Peiran Sun, Aleksandar Taranovic, Niklas Freymuth, Ge Li, Weiran Liao, C. F. Maximilian Nagy, Yucheng Tan, Tao Chen, Gerhard Neumann
Abstract:Visual imitation learning enables robots to acquire visuomotor skills directly from images, yet RGB observations lack explicit geometric cues, making learned policies brittle to camera perturbations. To address this, we propose \textbf{Ray-conditioned Vision Transformer Encoder (RayViT)}, a lightweight architecture that injects camera geometry into pretrained ViT backbones. RayViT represents camera geometry as a Plücker ray map, patchifies it into ray features, and uses gated cross-attention to produce a ray-conditioned class token. These ray features are added as dense positional embeddings, while the ray class token replaces the original ViT class token to provide a geometry-aware summary representation. We combine this approach with an auxiliary cosine similarity loss to consistently improve the performance and robustness for geometry-aware tokens. Experiments on sim- and real-robot tasks demonstrate that RayViT improves robustness by approximately 13 percentage points under camera perturbations in multi-task RoboCasa benchmark and by 1.78 average completed stages in real-world multi-task success rate compared to baselines.
Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2607.29622 [cs.RO]
(or arXiv:2607.29622v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2607.29622 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 201 words
Related coverage
G$^3$VLA: Geometric Inductive Bias for Vision-Language-Action Models
arXiv · Oct 15, 2026
ICLR: In-Context Imitation Learning with Visual Reasoning
arXiv · Jul 21, 2026
GuidedAttention: Interpretable and Correctable Visual Attention for OOD-Robust Robot Manipulation via Imitation Learning
arXiv · Jul 24, 2026
ContextFlow: In-Context Flow Matching for Robot Manipulation
arXiv · Sep 9, 2026