aiqbench analysis
This paper provides a rigorous geometric analysis of cosine similarity's limitations in embedding spaces, distinguishing between nuisance radial variation and critical angular distortions like hubness. It argues that while scale-invariance is often valid, practitioners must validate scoring rules against specific downstream decisions rather than applying the metric universally. The work offers concrete alternatives for scenarios where standard normalization discards essential information or fails to capture complex relational structures.
Semantics at an Angle: When Cosine Similarity Works Until It Doesn't
arXiv:2504.16318v3 Announce Type: replace Abstract: Cosine similarity is a standard comparison rule for learned representations in information retrieval, natural language processing, computer vision, and multimodal learning. Its popularity is well founded: it removes positive radial scale, is computationally convenient, and often matches objectives that train normalized embeddings. These same properties also delimit what cosine can express. Normalization discards radial variation; anisotropic representations can compress angular contrast; high-dimensional neighborhoods can develop hubs; and a symmetric, uncalibrated score may be mismatched to the relation of interest. This article offers a selective review organized around a simple principle: the usefulness of cosine similarity depends jointly on the learned representation, any normalization or post-processing, the scoring rule, and the downstream decision. We derive the main geometric identities, distinguish failure mechanisms that are often conflated, review representative evidence about embedding norms, and describe objective-matched, geometry-aware, hubness-aware, norm-aware, and learned alternatives. The central conclusion is conditional rather than adversarial: cosine's positive-scale invariance is justified when radial variation is nuisance or fixed by the representation contract, but its angular geometry and downstream decision must still be validated.
Abstract:Cosine similarity is a standard comparison rule for learned representations in information retrieval, natural language processing, computer vision, and multimodal learning. Its popularity is well founded: it removes positive radial scale, is computationally convenient, and often matches objectives that train normalized embeddings. These same properties also delimit what cosine can express. Normalization discards radial variation; anisotropic representations can compress angular contrast; high-dimensional neighborhoods can develop hubs; and a symmetric, uncalibrated score may be mismatched to the relation of interest. This article offers a selective review organized around a simple principle: the usefulness of cosine similarity depends jointly on the learned representation, any normalization or post-processing, the scoring rule, and the downstream decision. We derive the main geometric identities, distinguish failure mechanisms that are often conflated, review representative evidence about embedding norms, and describe objective-matched, geometry-aware, hubness-aware, norm-aware, and learned alternatives. The central conclusion is conditional rather than adversarial: cosine's positive-scale invariance is justified when radial variation is nuisance or fixed by the representation contract, but its angular geometry and downstream decision must still be validated.
Machine Learning (cs.LG)
arXiv:2504.16318 [cs.LG]
(or arXiv:2504.16318v3 [cs.LG] for this version)
Full text under the publisher's open licence · 186 words
Related coverage
Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
Hugging Face · Aug 18, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
Selecting a vector store for Amazon Bedrock Knowledge Bases
AWS · Sep 17, 2026
Presentation: Postgres for Production Agents: Your Relational Foundation for Enterprise AI
InfoQ · Jul 15, 2026