aiqbench analysis
This paper introduces a distributed conversational framework that integrates local vision-language models with a ROS 2 execution stack to enable responsive human-robot collaborative manipulation. By separating language understanding, visual grounding, and motion execution into distinct nodes, the system allows for flexible deployment while maintaining a safety-critical control loop that requires explicit operator confirmation before actuation. The approach demonstrates practical viability for complex pick-and-place tasks by converting free-form commands into structured robot actions under varying scene ambiguities.
A Conversational Framework for Human-Robot Collaborative Manipulation with Distributed Generative AI models
arXiv:2606.06061v1 Announce Type: new Abstract: This paper presents a distributed conversational framework for human-robot collaborative manipulation that integrates local language and vision-language models (VLMs) with a Robot Operating System 2 (ROS 2)-based execution stack. Language understanding, visual grounding, orchestration, and motion execution run as separate ROS 2 nodes, enabling flexible deployment across distributed hardware while maintaining a responsive control loop. From free-form user commands, the system generates structured action requests for pick, place, and handover. It uses a VLM to return image-space targets, which are converted into metric robot-frame goals using depth and calibration. A web dashboard exposes intermediate intent and grounding overlays (pixel, depth, and robot-frame) and requires explicit operator confirmation before any motion is executed. Experiments on a Franka FR3 platform evaluate end-to-end task reliability and latency under increasing working table scene ambiguity and compare alternative LLM/VLM configurations in the same pipeline. Code and full documentation are available at [github.com/cogrob-tuni/franka-llm](https://github.com/cogrob-tuni/franka-llm).
Abstract:This paper presents a distributed conversational framework for human-robot collaborative manipulation that integrates local language and vision-language models (VLMs) with a Robot Operating System 2 (ROS 2)-based execution stack. Language understanding, visual grounding, orchestration, and motion execution run as separate ROS 2 nodes, enabling flexible deployment across distributed hardware while maintaining a responsive control loop. From free-form user commands, the system generates structured action requests for pick, place, and handover. It uses a VLM to return image-space targets, which are converted into metric robot-frame goals using depth and calibration. A web dashboard exposes intermediate intent and grounding overlays (pixel, depth, and robot-frame) and requires explicit operator confirmation before any motion is executed. Experiments on a Franka FR3 platform evaluate end-to-end task reliability and latency under increasing working table scene ambiguity and compare alternative LLM/VLM configurations in the same pipeline. Code and full documentation are available at [this http URL](this https URL).
Accepted to the 35th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN 2026). The final published version will appear under the title "A Distributed Conversational Framework for Human-Robot Collaborative Manipulation Using Local LLMs and VLMs"
Robotics (cs.RO)
arXiv:2606.06061 [cs.RO]
(or arXiv:2606.06061v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2606.06061 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 205 words
Related coverage
How Reliable Are Multimodal Signals of Conversational State? Evidence from Remote Dyadic Collaborative Tasks
arXiv · Jul 21, 2026
Evaluating LLM-Simulated Conversations in Modeling Inconsistent and Uncollaborative Behaviors in Human Social Interaction
arXiv · Sep 11, 2026
CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement
arXiv · Jun 6, 2026
You Know What I Mean: A Benchmark for Agentic Conversational Reference Grounding
arXiv · Sep 1, 2026