aiqbench analysis
This research exposes a critical gap in encoder-based transformer models like DeBERTa, which fail to capture pragmatic distinctions between lexical and periphrastic causatives despite their strong semantic capabilities. While the M-Heuristic remains unavailable in default Natural Language Inference tasks for these architectures, it can be activated via explicit metalinguistic framing in larger models like Gemini Flash-Lite. These findings highlight the need for further investigation into how current NLP systems handle neo-Gricean principles and whether such limitations represent a fundamental architectural constraint or a training data issue.
Larry Caused the Car to Stop, But the Model Didn't Notice: Transformer Blindness to the M-Heuristic
arXiv:2609.37497v1 Announce Type: new Abstract: Modern transformer models excel at capturing semantic relationships through sentence embeddings, yet their ability to perform pragmatic reasoning remains understudied. This paper investigates whether encoder-based transformers such as DeBERTa employ the M-Heuristic (the neo-Gricean principle that marked linguistic forms implicate marked meanings). We test this hypothesis by contrasting lexical causatives (e.g., ``Larry stopped the car'') with periphrastic causatives (e.g., ``Larry caused the car to stop'') using a Natural Language Inference framework. Our experiments across 188 conditions with 15 ambitransitive verbs reveal that DeBERTa, RoBERTa, and BART show no evidence of capturing the pragmatic distinction between these forms, with DeBERTa predicting ``Neutral'' for 100% of cases. Probing analysis initially suggested a representation-use dissociation, but control experiments reveal the probe was tracking syntactic complexity, not causative pragmatics. Semantic similarity over 30 triplets places periphrastic causatives closer to unmediated manner descriptions in 29/30 cases, opposite to M-Heuristic predictions in the embedding space. Under explicit metalinguistic framing, Gemini Flash-Lite reaches 100% with item-specific traces, so the principle is available under instruction yet unused in default NLI.
Abstract:Modern transformer models excel at capturing semantic relationships through sentence embeddings, yet their ability to perform pragmatic reasoning remains understudied. This paper investigates whether encoder-based transformers such as DeBERTa employ the M-Heuristic (the neo-Gricean principle that marked linguistic forms implicate marked meanings). We test this hypothesis by contrasting lexical causatives (e.g., ``Larry stopped the car'') with periphrastic causatives (e.g., ``Larry caused the car to stop'') using a Natural Language Inference framework. Our experiments across 188 conditions with 15 ambitransitive verbs reveal that DeBERTa, RoBERTa, and BART show no evidence of capturing the pragmatic distinction between these forms, with DeBERTa predicting ``Neutral'' for 100% of cases. Probing analysis initially suggested a representation-use dissociation, but control experiments reveal the probe was tracking syntactic complexity, not causative pragmatics. Semantic similarity over 30 triplets places periphrastic causatives closer to unmediated manner descriptions in 29/30 cases, opposite to M-Heuristic predictions in the embedding space. Under explicit metalinguistic framing, Gemini Flash-Lite reaches 100% with item-specific traces, so the principle is available under instruction yet unused in default NLI.
Computation and Language (cs.CL)
arXiv:2609.37497 [cs.CL]
(or arXiv:2609.37497v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.37497 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 267 words
