aiqbench analysis
GazeRefine introduces a novel inference-time prompting mechanism that leverages human eye-tracking data to drive zero-shot medical image segmentation without requiring model fine-tuning or extensive labeled datasets. By converting sparse gaze fixations into semantic prototypes within a frozen feature space, the method effectively balances segmentation accuracy with label efficiency in critical domains like polyp detection and prostate MRI analysis. This approach represents a significant step toward scalable, human-in-the-loop AI systems that minimize dependency on costly expert annotations.
GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation
arXiv:2609.01310v1 Announce Type: cross Abstract: Medical image segmentation remains difficult to scale because high-performing methods typically rely on dense expert annotations and task-specific training. We introduce GazeRefine, a training-free framework that uses gaze as an inference-time prompt for zero-shot medical image segmentation. Sparse, duration-weighted fixations are converted into foreground and background priors that initialize semantic prototypes in frozen DINOv3 feature space. These prototypes are iteratively refined through foreground-background discrimination, feature-space affinity propagation, and anchoring to the initial gaze guidance, allowing segmentation to extend beyond directly fixated regions while limiting semantic drift. GazeRefine requires no segmentation masks, fine-tuning, adapters, prompt encoders, or gradient updates. We evaluate the method on gaze-annotated polyp segmentation and prostate MRI segmentation. The results show strong performance on colonoscopy images and competitive performance on prostate MRI, supporting gaze-guided prototype refinement as a promising approach for segmentation-label-efficient, human-in-the-loop medical image segmentation. Our tools and code can be found in the following repository: https://github.com/MohammedOussamaBEN/GazeRefine.git
Abstract:Medical image segmentation remains difficult to scale because high-performing methods typically rely on dense expert annotations and task-specific training. We introduce GazeRefine, a training-free framework that uses gaze as an inference-time prompt for zero-shot medical image segmentation. Sparse, duration-weighted fixations are converted into foreground and background priors that initialize semantic prototypes in frozen DINOv3 feature space. These prototypes are iteratively refined through foreground-background discrimination, feature-space affinity propagation, and anchoring to the initial gaze guidance, allowing segmentation to extend beyond directly fixated regions while limiting semantic drift. GazeRefine requires no segmentation masks, fine-tuning, adapters, prompt encoders, or gradient updates. We evaluate the method on gaze-annotated polyp segmentation and prostate MRI segmentation. The results show strong performance on colonoscopy images and competitive performance on prostate MRI, supporting gaze-guided prototype refinement as a promising approach for segmentation-label-efficient, human-in-the-loop medical image segmentation. Our tools and code can be found in the following repository: this https URL
9 pages, 5 figures. Accepted at MICCAI Workshop 2026
Image and Video Processing (eess.IV); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG)
arXiv:2609.01310 [eess.IV]
(or arXiv:2609.01310v1 [eess.IV] for this version)
https://doi.org/10.48550/arXiv.2609.01310 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 196 words
