LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation

arXiv:2606.27871v2 Announce Type: replace Abstract: Vision Language Models (VLMs) have emerged in the robotic domain as a powerful tool that enables environmental perception with language context, serving as a catalyst for open- vocabulary tasks like Object Goal Navigation (ObjectNav). Yet, their computational footprint typically confines them to cloud execution, hindering low-latency inference with local deployment on resource-constrained robots. To address this challenge, we present a distillation strategy that aligns a lightweight, 4B- parameter local VLM to our spatial-semantic navigation interface via frontier model traces for edge execution on embedded GPU devices (e.g., Jetson Orin). While cloud models establish a baseline achieving 41.61% Success Rate (SR) on the HM3D OVON benchmark, zero-shot deployment of our proposed architecture with Qwen3.5-4B reaches only 25.97% SR. We demonstrate that distilling frontier reasoning traces and subsequent finetuning with Embodied Reinforcement Learning from Verifiable Rewards (E- RLVR) obtains a 37.36% SR, substantially narrowing the gap to large cloud models. However, while deployable on memory- constrained devices, the model cannot compete with round-trip cloud inference time: we identify Token Generation (TG) as the main driver of latency and reduce it with Token Generation (TG)- regularized E-RLVR. The combined method reduces latency by 69.29% and retains a SR of 35.53%. Combined with quantization, this joint strategy yields a cumulative 80.21% reduction in overall inference latency without significantly sacrificing performance, presenting a viable paradigm for local, low-latency VLM execu- tion on mobile robots.