IMAC-AgriVLN: Can Agricultural Vision-and-Language Navigation Agents be Aware of Instruction Mistakes?

Title:IMAC-AgriVLN: Can Agricultural Vision-and-Language Navigation Agents be Aware of Instruction Mistakes?

Abstract:Agricultural robots are playing as important roles across a wide range of tasks, nevertheless, they are still mainly depending on manual operations or fixed railways for moving. The A2A benchmark and the AgriVLN method pioneeringly extended Vision-and-Language Navigation (VLN) to the agricultural domain, successfully navigating agricultural robots from starting points to target positions following natural language instructions. However, we observed that almost all the prior VLN methods adopted an ideal assumption: The given instructions themselves were correct. This assumption did not align with the realistic scenarios, because anybody might say an instruction with mistakes, which raised us a question: If an instruction had a mistake, could an agricultural VLN agent be aware of it? To answer this question, we propose the A2A-MI benchmark, in which we follow A2A as the foundation benchmark and insert three classes of instruction mistakes. We use it to evaluate several state-of-the-art agricultural VLN agents, then observe sufficient drops across all of them, such as AgriVLN decreases SR by 57% in average and increases NE by 9% in average, from which we suggest the lacking awareness on instruction mistakes. To address this problem, we propose the IMAC module analyzing the instruction and image, to reason whether the instruction has mistakes and attempt to correct them when needed. We integrate it into the AgriVLN backbone to build our IMAC-AgriVLN method, successfully saving SR from 0.10 to 0.14 and NE from 4.81 m to 4.79 m, which demonstrates the effectiveness of IMAC on strengthening the robustness against instruction mistakes. Project: this https URL.

Robotics (cs.RO)

arXiv:2606.02519 [cs.RO]

(or arXiv:2606.02519v3 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2606.02519 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.