analiza aiqbench
This paper addresses a practical bottleneck in training robot imitation-learning policies: real-world evaluation is costly and sparse, while plain validation loss often fails to predict rollout performance. The proposed CI-MSE metric restricts error to task-critical segments identified by a zero-shot agent, and it reports a Spearman correlation of -0.87 across 27 checkpoints versus -0.61 for raw MSE. Readers should note that these results come from one set of simulation and real-world experiments, and the stated limits on distribution shift matter for whether the metric transfers to their own setups.
Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policies
arXiv:2606.29898v2 Announce Type: replace-cross Abstract: Real-world evaluation is the gold standard for robot manipulation policies because it tests them against the physical conditions and deployment challenges they are ultimately designed to handle. However, real-world evaluation is also the bottleneck for iterating on robot imitation learning policies: it is costly, difficult to reproduce, and often too sparse to reliably compare nearby model variants. A straightforward proxy for performance is validation loss on expert demonstrations, but this proxy is often poorly correlated with real-world performance. In this paper, we introduce Critical Interval MSE (CI-MSE), an intuitively simple yet effective offline validation metric. CI-MSE uses a zero-shot agent to localize task-critical segments, restricts error computation to those segments, and applies simple action-alignment procedures that better match rollout-time behavior. Across simulation and real-world experiments, CI-MSE generally yields a stronger correlation between validation error and rollout performance than raw MSE. Across 27 distinct policy checkpoints, CI-MSE achieves a Spearman rank correlation of $-0.87$, closer to the ideal value of $-1$ than raw MSE's $-0.61$, demonstrating a significant improvement. Sensitivity analysis shows that our metric is robust across a wide range of hyperparameter settings. We further study the effectiveness of CI-MSE under evaluation distribution shifts and identify practical boundaries for its use. In summary, this paper provides a simple and reliable offline validation tool for accelerating policy iteration. Project webpage: https://ci-mse.github.io/
Abstract:Real-world evaluation is the gold standard for robot manipulation policies because it tests them against the physical conditions and deployment challenges they are ultimately designed to handle. However, real-world evaluation is also the bottleneck for iterating on robot imitation learning policies: it is costly, difficult to reproduce, and often too sparse to reliably compare nearby model variants. A straightforward proxy for performance is validation loss on expert demonstrations, but this proxy is often poorly correlated with real-world performance. In this paper, we introduce Critical Interval MSE (CI-MSE), an intuitively simple yet effective offline validation metric. CI-MSE uses a zero-shot agent to localize task-critical segments, restricts error computation to those segments, and applies simple action-alignment procedures that better match rollout-time behavior. Across simulation and real-world experiments, CI-MSE generally yields a stronger correlation between validation error and rollout performance than raw MSE. Across 27 distinct policy checkpoints, CI-MSE achieves a Spearman rank correlation of $-0.87$, closer to the ideal value of $-1$ than raw MSE's $-0.61$, demonstrating a significant improvement. Sensitivity analysis shows that our metric is robust across a wide range of hyperparameter settings. We further study the effectiveness of CI-MSE under evaluation distribution shifts and identify practical boundaries for its use. In summary, this paper provides a simple and reliable offline validation tool for accelerating policy iteration. Project webpage: this https URL
Robotics (cs.RO); Artificial Intelligence (cs.AI)
arXiv:2606.29898 [cs.RO]
(or arXiv:2606.29898v2 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2606.29898 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 316 cuvinte
