aiqbench analysis
This study identifies lexical ambiguity resolution as a core bottleneck for large language models, demonstrating that explicit sense inventories significantly improve performance over implicit context-only approaches. The findings reveal that current model errors often stem from mismatched semantic granularity between annotators and the model rather than a fundamental lack of understanding. Consequently, future advancements in natural language processing will likely depend on aligning training data with standardized sense boundaries to reduce over-refinement issues.
WiC is Not WSD: A Study on LLMs and Lexical Ambiguity Resolution
arXiv:2609.20593v1 Announce Type: new Abstract: Word-in-Context (WiC) remains challenging for language models, despite recent progress on lexical-semantic tasks. We hypothesise that this difficulty arises not only from comparing two contextual uses of a word, but also from the absence of an explicit sense inventory that specifies the relevant level of semantic granularity. We evaluate open LLMs on WiC and traditional Word Sense Disambiguation (WSD) under similar settings. We find that providing candidate senses, similar to what is done in traditional WSD, improves WiC performance in all settings. In general, explicit sense information helps models make more consistent and targeted judgements. Human evaluation further shows that many apparent WiC errors reflect label ambiguity or mismatches between model and annotator sense boundaries rather than simple failures of lexical understanding. In particular, results show that LLMs overthink the sense distinction often leading to errors based on overly fine-grained distinctions.
Abstract:Word-in-Context (WiC) remains challenging for language models, despite recent progress on lexical-semantic tasks. We hypothesise that this difficulty arises not only from comparing two contextual uses of a word, but also from the absence of an explicit sense inventory that specifies the relevant level of semantic granularity. We evaluate open LLMs on WiC and traditional Word Sense Disambiguation (WSD) under similar settings. We find that providing candidate senses, similar to what is done in traditional WSD, improves WiC performance in all settings. In general, explicit sense information helps models make more consistent and targeted judgements. Human evaluation further shows that many apparent WiC errors reflect label ambiguity or mismatches between model and annotator sense boundaries rather than simple failures of lexical understanding. In particular, results show that LLMs overthink the sense distinction often leading to errors based on overly fine-grained distinctions.
Computation and Language (cs.CL)
arXiv:2609.20593 [cs.CL]
(or arXiv:2609.20593v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.20593 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 236 words
Related coverage
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding
arXiv · Sep 30, 2026
FrenchNews-7: Benchmarking Cross-Publisher French News Editorial Desk Classification
arXiv · Aug 20, 2026
We're Cooked! - Probing LLM Political Alignment Via Conflict-Framed Recipe Translation
arXiv · Sep 9, 2026