analiza aiqbench
This research exposes a critical fragility in Vision-Language-Action (VLA) models, revealing that they rely heavily on surface-level instruction matching rather than true semantic grounding for robotic tasks. The study demonstrates that paraphrasing instructions causes significant performance degradation and planning failures across various model sizes, necessitating new metrics like PRIDE to better evaluate linguistic robustness.
LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models
arXiv:2603.28301v2 Announce Type: replace Abstract: Vision-Language-Action (VLA) models achieve strong performance in robotic manipulation by leveraging pre-trained vision-language backbones. However, in downstream robotic settings, they are typically fine-tuned with limited data, leading to overfitting to specific instruction formulations and leaving robustness to paraphrased instructions underexplored. To study this gap, we introduce LIBERO-Para, a controlled benchmark that independently varies action expressions and object references for fine-grained analysis of linguistic generalization. Across seven VLA configurations (0.6B-7.5B), we observe consistent performance degradation of 22-52 pp under paraphrasing. This degradation is primarily driven by object-level lexical variation: even simple synonym substitutions cause large drops, indicating reliance on surface-level matching rather than semantic grounding. Moreover, 80-96% of failures arise from planning-level trajectory divergence rather than execution errors, showing that paraphrasing disrupts task identification. Binary success rate treats all paraphrases equally, obscuring whether models perform consistently across difficulty levels or rely on easier cases. To address this, we propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors. Our benchmark and corresponding code are available at: https://github.com/cau-hai-lab/LIBERO-Para
Abstract:Vision-Language-Action (VLA) models achieve strong performance in robotic manipulation by leveraging pre-trained vision-language backbones. However, in downstream robotic settings, they are typically fine-tuned with limited data, leading to overfitting to specific instruction formulations and leaving robustness to paraphrased instructions underexplored. To study this gap, we introduce LIBERO-Para, a controlled benchmark that independently varies action expressions and object references for fine-grained analysis of linguistic generalization. Across seven VLA configurations (0.6B-7.5B), we observe consistent performance degradation of 22-52 pp under paraphrasing. This degradation is primarily driven by object-level lexical variation: even simple synonym substitutions cause large drops, indicating reliance on surface-level matching rather than semantic grounding. Moreover, 80-96% of failures arise from planning-level trajectory divergence rather than execution errors, showing that paraphrasing disrupts task identification. Binary success rate treats all paraphrases equally, obscuring whether models perform consistently across difficulty levels or rely on easier cases. To address this, we propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors. Our benchmark and corresponding code are available at: this https URL
Accepted to EMNLP 2026 (Main Conference). Code and benchmark: this https URL
Machine Learning (cs.LG)
arXiv:2603.28301 [cs.LG]
(or arXiv:2603.28301v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2603.28301 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 275 cuvinte
Subiecte conexe
IndustrialVLA-Bench: A Traceable Multi-Axis Evaluation of Open Robot Policy Models
arXiv · 24 sept. 2026
How to Evaluate General-Purpose Robot Policies for Real-World Deployment
NVIDIA · 12 iul. 2026
Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models
arXiv · 4 aug. 2026
Does Continual Imitation Learning Remain Grounded? A Language-Perturbed Benchmark for Robotic Task Retention
arXiv · 2 oct. 2026