aiqbench analysis
The paper evaluates how robot "world-action" policies respond to different wordings of the same spatial goal, using 864 rollouts across three models. It finds that phrasing a relation through the reference object instead of the object to be moved lowers stable placement by 12.2 points on average, and that aggregate scores can mask this sensitivity. For AI readers, the more useful point is the evaluation method: checking instruction consistency, verifying the correct object moved, and testing whether placements persist to episode end.
Same Goal, Different Words: Spatial Instruction Sensitivity in World-Action Models
arXiv:2610.16242v1 Announce Type: new Abstract: World-action models should preserve the intended manipulation across equivalent descriptions of a spatial goal. We evaluate this consistency in 864 rollouts from three models across four RoboLab scenes, holding initial states and task goals fixed. Describing relations through the reference object rather than the object-to-move lowers the rate of reaching a stable placement by 12.2 percentage points on average. The effect of instruction rephrasing varies across spatial relations, so aggregate scores can hide sensitivity to wording. Standalone tests of the policies' language understanding identify relation reversals and show that adding images can disrupt otherwise correct text-only interpretations. Rollout analysis also distinguishes spatial placement from instruction following: moving the wrong object can satisfy the requested relation, and rates of stable placement fall from 43.2% at any time to 21.8% at episode end. These findings motivate a more thorough evaluation of world-action models' steerability, language understanding, and grounding. Evaluations should test consistency across equivalent instructions, verify that the intended object moved, and check whether the requested arrangement persists to the end of the rollout.
Abstract:World-action models should preserve the intended manipulation across equivalent descriptions of a spatial goal. We evaluate this consistency in 864 rollouts from three models across four RoboLab scenes, holding initial states and task goals fixed. Describing relations through the reference object rather than the object-to-move lowers the rate of reaching a stable placement by 12.2 percentage points on average. The effect of instruction rephrasing varies across spatial relations, so aggregate scores can hide sensitivity to wording. Standalone tests of the policies' language understanding identify relation reversals and show that adding images can disrupt otherwise correct text-only interpretations. Rollout analysis also distinguishes spatial placement from instruction following: moving the wrong object can satisfy the requested relation, and rates of stable placement fall from 43.2% at any time to 21.8% at episode end. These findings motivate a more thorough evaluation of world-action models' steerability, language understanding, and grounding. Evaluations should test consistency across equivalent instructions, verify that the intended object moved, and check whether the requested arrangement persists to the end of the rollout.
19 pages, including appendices. Corrected the spelling and affiliation of Taşkın Padır and added his institutional disclaimer. Scientific content unchanged
Robotics (cs.RO)
arXiv:2610.16242 [cs.RO]
(or arXiv:2610.16242v2 [cs.RO] for this version)
Full text under the publisher's open licence · 202 words
Related coverage
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026