aiqbench analysis
This arXiv paper addresses a practical problem in open-vocabulary 3D segmentation: CLIP features lifted into 3D Gaussian reconstructions cluster in a scene-specific direction, which hurts raw cosine-similarity matching. The authors show that training-free, query-time adjustments (centering by a shared scene direction, whitening text prototypes against a reference vocabulary, and CSLS-style scoring) raise mIoU across seven lifting methods and four datasets. The main takeaway for practitioners is that existing lifted representations may be underused, and that comparison-side calibration can be a low-cost gain, though the evidence is limited to benchmark segmentation gains.
Query-Time Calibration of CLIP Features for Open-Vocabulary 3D Segmentation
arXiv:2610.30908v1 Announce Type: cross Abstract: Lifting vision-language features such as CLIP into 3D Gaussian reconstructions enables open-vocabulary segmentation through natural-language queries. Although CLIP represents a broad range of semantic concepts, the lifted features of individual scenes can exhibit strong directional concentration, while raw cosine similarity does not account for the scene-specific feature distribution. We show that substantial segmentation gains can be obtained by changing how a fixed lifted feature field is compared with text. Our method partially subtracts a shared scene direction from visual features and text prototypes and whitens text prototypes using a fixed reference vocabulary of generic terms. It then uses soft clustering for multiclass prediction and CSLS with a relevance margin against the reference vocabulary for single-query prediction. Beyond reducing directional concentration, scene centering improves class-relative feature separation in every evaluated scene. Across seven lifting methods and four datasets, the resulting query-time calibration consistently improves mean intersection over union in both multiclass and single-query segmentation. These gains require no retraining, source-image re-encoding, or feature re-lifting, demonstrating the value of scene-conditioned comparison for making better use of existing lifted representations.
Read the full article at arXiv