Uncertainty-Aware Action Segmentation for Human-to-Robot Assembly Teleoperation

Abstract:Action segmentation supports intention estimation and execution monitoring in assisted teleoperation, but robot demonstrations are costly to collect. We propose an uncertainty-aware framework that transfers temporal knowledge from human to robot demonstrations through MS-TCN++~\cite{li2020mstcnpp} pretraining and fine-tuning, reliability-aware sequence correction, and class-conditional conformal prediction. Across 22 toy-car assembly action classes, fine-tuning with 9 robot demonstrations improves test Edit score from 55.23 to 72.51 and segmental F1@50 from 10.97 to 16.86. The reliability analyzer identifies incorrect segments with 0.755 AUROC. Unrestricted correction reduces Edit to 67.86, while selective acceptance maintains 73.45, primarily safeguarding against harmful relabeling. Conformal prediction achieves 95.38\% empirical coverage at a 93\% target, but prediction sets average 17.41 of 22 classes. These findings support human temporal pretraining while revealing persistent challenges in reliable correction and informative uncertainty quantification with limited robot supervision.

7 pages, 6 figures. Preprint version

Robotics (cs.RO)

arXiv:2606.08341 [cs.RO]

(or arXiv:2606.08341v2 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2606.08341 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.