Multi-modal Representation Fusion for Long-term Microsurgical Anastomosis Trajectory Anticipation

Abstract:Microsurgical anastomosis is one of the most challenging and sophisticated procedures for surgical robotics, requiring accurate manipulation of surgical instruments within a limited space to achieve precise operations. Accurate surgical trajectory anticipation is essential for surgical scene understanding, intraoperative planning, robotic manipulation, and reliable behaviour analysis. However, existing approaches mainly focus on features extracted from surgical videos and historical trajectories, while overlooking rich semantic and motion information that can provide more discriminative multi-modal representations for complex surgical instrument-tissue interactions. In this work, we propose a cross-modal feature fusion framework for long-term 2D and 3D microsurgical anastomosis trajectory anticipation. The proposed framework consists of a multi-modal representation extractor and a cross-modal feature fusion network. Semantic recognition of surgical objects and consecutive motion estimation are incorporated as multi-modal visual cues, providing complementary spatial and temporal guidance for instrument-tissue interaction characterization. A coarse-to-fine progressive fusion strategy with a long-short-term memory mechanism is introduced to capture temporal dependencies step by step. The training and evaluation are conducted on our established robotic platform and a comprehensive dataset containing both phantom and ex-vivo surgical scenarios is constructed. Extensive quantitative and qualitative experiments demonstrate that the proposed framework consistently outperforms existing methods and exhibits improved robustness across different surgical scenes, highlighting its potential for both intelligent robot-assisted and autonomous surgical systems.

Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)

arXiv:2610.14492 [cs.RO]

(or arXiv:2610.14492v1 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2610.14492 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.