analiza aiqbench
This study identifies four structural principles governing concept representation in sentence encoders, revealing that fine-tuning recalibrates latent geometry rather than expanding it and that semantic signals concentrate in the final transformer layer. The research exposes a critical limitation where extensional training paradigms degrade performance on relational and intensional concepts, suggesting a need for composition-type-aware supervision strategies. Additionally, the authors release new evaluation datasets to benchmark these findings, offering a rigorous framework for assessing encoder robustness independent of retrieval ranking metrics.
Principles of Concept Representation in Sentence Encoders
arXiv:2606.06994v1 Announce Type: new Abstract: What makes a sentence encoder produce good concept representations? We approach this through the lens of representational compositionality: an encoder supports a concept family only when its latent space admits a low-distortion realization of the corresponding semantic operator. This framing predicts both where current encoders succeed and where they are structurally mismatched to their supervision. Through a controlled ablation over encoder conditions trained on 3.3 million synonym and definition pairs from WordNet and Wiktionary, evaluated on three decontaminated splits and a modifier-labeled noun-phrase benchmark, we identify four principles. Fine-tuning recalibrates the latent geometry rather than expanding it (P1). Semantic signal concentrates in the final transformer layer before concept-specific training begins, making cross-layer pooling redundant (P2). Hard negatives improve discrimination and stress-test robustness without improving retrieval ranking, showing that calibration and ranking are independently addressable (P3). Finally, the effectiveness of supervision depends on the composition type of the target concept. Extensional training helps intersective and subsective families while degrading relational and intensional ones, exposing a structural limitation of current training paradigms (P4). We release two new evaluation datasets: a DBpedia semantic-gap benchmark and a modifier-labeled NP paraphrase suite.
Abstract:What makes a sentence encoder produce good concept representations? We approach this through the lens of representational compositionality: an encoder supports a concept family only when its latent space admits a low-distortion realization of the corresponding semantic operator. This framing predicts both where current encoders succeed and where they are structurally mismatched to their supervision. Through a controlled ablation over encoder conditions trained on 3.3 million synonym and definition pairs from WordNet and Wiktionary, evaluated on three decontaminated splits and a modifier-labeled noun-phrase benchmark, we identify four principles. Fine-tuning recalibrates the latent geometry rather than expanding it (P1). Semantic signal concentrates in the final transformer layer before concept-specific training begins, making cross-layer pooling redundant (P2). Hard negatives improve discrimination and stress-test robustness without improving retrieval ranking, showing that calibration and ranking are independently addressable (P3). Finally, the effectiveness of supervision depends on the composition type of the target concept. Extensional training helps intersective and subsective families while degrading relational and intensional ones, exposing a structural limitation of current training paradigms (P4). We release two new evaluation datasets: a DBpedia semantic-gap benchmark and a modifier-labeled NP paraphrase suite.
Computation and Language (cs.CL); Databases (cs.DB)
arXiv:2606.06994 [cs.CL]
(or arXiv:2606.06994v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.06994 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 284 cuvinte
Subiecte conexe
InfoNCE Induces Gaussian Distribution
arXiv · 16 iun. 2026
Realize What Matters: Principled Context Representation for Large-Scale Reasoning
arXiv · 24 sept. 2026
Twice Upon a Time: Complementary Representations for Time-Series Reasoning
arXiv · 17 oct. 2026
Solving and Learning Navier-Stokes Dynamics with the Neural Basis Method
arXiv · 27 oct. 2026