analiza aiqbench
This arXiv paper argues that embedding similarity, the standard way to measure LLM homogeneity, collapses too much. Using 25 interpretable metrics across five dimensions, the authors find that models often convey similar content in different wording, that their intra-model behavior is converging across releases, and that their inter-model differences are growing. For researchers evaluating model diversity or training pipelines, the practical point is that a single similarity score can overstate sameness and hide trends in form and style.
Revisiting LLM Homogeneity Through a Multidimensional Lens
arXiv:2611.01549v1 Announce Type: cross Abstract: Concerns about LLM homogeneity are typically quantified with embedding similarity, which reduces two texts to a single score. We revisit homogeneity with 25 interpretable metrics spanning five dimensions: semantic, lexical, syntactic, structural, and stylistic & presentational. A human study shows that perceived similarity depends on both content (what responses say) and form (how they say it), yet embedding similarity correlates less with human meaning judgments than sentence-level matching ($\rho=0.417$ vs. $0.606$) and not significantly with form judgments ($\rho=0.140$). Across 18 contemporary LLMs, models say similar things in different words: they share global semantics and broad linguistic profiles but remain distinct in lexical characteristics, structural patterns, and interactional conventions. Across releases from 2024 to 2026, models become more like themselves but not more like one another: among significantly changing metrics, intra-model distances decrease for 17 of 22, and inter-model distances increase for 14 of 21. Across two post-training pipelines, model-specific conventions emerge during SFT and are selectively reshaped by preference optimization. These results suggest that homogeneity is multidimensional and evolving, and that embedding similarity alone can overstate cross-model homogeneity while concealing diverging intra- and inter-model trends.
Abstract:Concerns about LLM homogeneity are typically quantified with embedding similarity, which reduces two texts to a single score. We revisit homogeneity with 25 interpretable metrics spanning five dimensions: semantic, lexical, syntactic, structural, and stylistic & presentational. A human study shows that perceived similarity depends on both content (what responses say) and form (how they say it), yet embedding similarity correlates less with human meaning judgments than sentence-level matching ($\rho=0.417$ vs. $0.606$) and not significantly with form judgments ($\rho=0.140$). Across 18 contemporary LLMs, models say similar things in different words: they share global semantics and broad linguistic profiles but remain distinct in lexical characteristics, structural patterns, and interactional conventions. Across releases from 2024 to 2026, models become more like themselves but not more like one another: among significantly changing metrics, intra-model distances decrease for 17 of 22, and inter-model distances increase for 14 of 21. Across two post-training pipelines, model-specific conventions emerge during SFT and are selectively reshaped by preference optimization. These results suggest that homogeneity is multidimensional and evolving, and that embedding similarity alone can overstate cross-model homogeneity while concealing diverging intra- and inter-model trends.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2611.01549 [cs.CL]
(or arXiv:2611.01549v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.01549 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 283 cuvinte
Subiecte conexe
A convolutional autoencoder and neural ODE surrogate modeling framework applied to transient counterflow flames
arXiv · 10 sept. 2026
CANN-EUCLID: unsupervised constitutive artificial neural network model discovery from full-field data
arXiv · 15 iun. 2026
Wisdom in Unity: The Role of Multilingual Training in Figurative Language Identification in Proverbs
arXiv · 24 sept. 2026