analiza aiqbench
This paper proposes a way to segment robot demonstrations by feeding a language model compact, learned tokens built from joint-level kinematic data, instead of raw multi-camera video. The approach reports better F1 than baselines and zero-shot transfer to new robot embodiments and human motion, but these are arXiv preprint results on human-labeled benchmarks, so their robustness in production settings is still unproven. For ML readers, the useful idea is discrete motion vocabularies that cut MLLM context costs for robot data.
KiRoSe: Using Kinematic Data for Efficient Segmentation of Robot Demonstrations
arXiv:2610.31801v1 Announce Type: new Abstract: Segmenting robot demonstration data into disjoint temporal and semantic subsegments is valuable for learning, cleaning, retrieval, and analysis. Segmentation is often performed manually in a tedious and error prone process. Multimodal Large Language Models (MLLMs) have the potential for automating this process, but directly ingesting raw multi camera video streams into MLLMs can be costly and can quickly exhaust context windows. In this paper, we introduce KiRoSe (Kinematics for Robot Segmentation), a novel framework that focuses on the companion stream of robot kinematic data. Inspired by the principle of classical entropy coding, such as Huffman coding, where frequently occurring symbols are represented more compactly, we propose Sparse Motion Tokens (SMTs), a discrete kinematic vocabulary designed to compactly represent recurring, temporally localized motion patterns, which we refer to as kinematic events. SMTs are learned by encoding kinematic trajectories with temporal segment supervision and quantizing the resulting features. We use increasingly available robot demonstration subtask annotations to supervise an open language model to interpret the resulting token sequences. Evaluation against human labeled segmentation shows that KiRoSe can substantially outperform baseline segmentation methods in F1 quality and transfers zero shot to unseen robot embodiments and egocentric human motion, while preserving the language models open vocabulary knowledge. Beyond segmentation, KiRoSe tracks task progress on an unseen robot and detects when an action is being reversed.(website: https://berkeleyautomation.github.io/kirose/)
Abstract:Segmenting robot demonstration data into disjoint temporal and semantic subsegments is valuable for learning, cleaning, retrieval, and analysis. Segmentation is often performed manually in a tedious and error prone process. Multimodal Large Language Models (MLLMs) have the potential for automating this process, but directly ingesting raw multi camera video streams into MLLMs can be costly and can quickly exhaust context windows. In this paper, we introduce KiRoSe (Kinematics for Robot Segmentation), a novel framework that focuses on the companion stream of robot kinematic data. Inspired by the principle of classical entropy coding, such as Huffman coding, where frequently occurring symbols are represented more compactly, we propose Sparse Motion Tokens (SMTs), a discrete kinematic vocabulary designed to compactly represent recurring, temporally localized motion patterns, which we refer to as kinematic events. SMTs are learned by encoding kinematic trajectories with temporal segment supervision and quantizing the resulting features. We use increasingly available robot demonstration subtask annotations to supervise an open language model to interpret the resulting token sequences. Evaluation against human labeled segmentation shows that KiRoSe can substantially outperform baseline segmentation methods in F1 quality and transfers zero shot to unseen robot embodiments and egocentric human motion, while preserving the language models open vocabulary knowledge. Beyond segmentation, KiRoSe tracks task progress on an unseen robot and detects when an action is being reversed.(website: this https URL)
Robotics (cs.RO)
arXiv:2610.31801 [cs.RO]
(or arXiv:2610.31801v1 [cs.RO] for this version)
https://doi.org/10.48550/arXiv.2610.31801 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 317 cuvinte
Subiecte conexe
SARTM: Segment Any RGB Thermal Model with Language aided Distillation
arXiv · 2 sept. 2026
An Empirical Study of Wrist-Target Construction and Joint-Angle Limits in Humanoid Arm Retargeting
arXiv · 30 oct. 2026
Evaluation of Blood Vessel Segmentation Methods on Hard-to-Detect Vascular Structures
arXiv · 28 iul. 2026
Superpixel-Based QUBO for Scalable Quantum-Enhanced Medical Image Segmentation
arXiv · 28 iul. 2026