analiza aiqbench
This study challenges the assumed utility of Abstract Meaning Representation (AMR) augmentation for modern large language models, demonstrating that text-only baselines often match or exceed performance when experimental protocols are standardized. By introducing a perplexity-based probe, the authors reveal that LLMs already possess sufficient relational knowledge internally, rendering external AMR graphs unnecessary for downstream tasks. This finding suggests a shift away from structural graph augmentations toward optimizing base model architectures and training data for relation understanding.
On the (In)effectiveness of AMR Augmentation for Large Language Models
arXiv:2609.40121v1 Announce Type: cross Abstract: While Abstract Meaning Representation (AMR) has historically improved performance on a range of NLP tasks, the benefit---or lack thereof---of AMR augmentation for modern LLMs is thus far unclear. In this paper, we attempt to reproduce recent work that reported substantial downstream gains from AMR augmentation, finding that these are likely due to specific choices in the experimental settings used: using a consistent and unified protocol for hyperparameter selection, we observe that text-only baselines consistently match or exceed the performance of AMR-augmented models. To investigate this null result, we introduce a perplexity-based probe measuring the degree to which AMR provides an LLM with supplemental relational knowledge not already available to the model. We find that AMR augmentation does not help LLMs improve their understanding of relational content in the sentence, indicating that augmenting these models with AMR offers no clear benefit on downstream tasks.
Abstract:While Abstract Meaning Representation (AMR) has historically improved performance on a range of NLP tasks, the benefit---or lack thereof---of AMR augmentation for modern LLMs is thus far unclear. In this paper, we attempt to reproduce recent work that reported substantial downstream gains from AMR augmentation, finding that these are likely due to specific choices in the experimental settings used: using a consistent and unified protocol for hyperparameter selection, we observe that text-only baselines consistently match or exceed the performance of AMR-augmented models. To investigate this null result, we introduce a perplexity-based probe measuring the degree to which AMR provides an LLM with supplemental relational knowledge not already available to the model. We find that AMR augmentation does not help LLMs improve their understanding of relational content in the sentence, indicating that augmenting these models with AMR offers no clear benefit on downstream tasks.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2609.40121 [cs.CL]
(or arXiv:2609.40121v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.40121 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 241 cuvinte
Subiecte conexe
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool
AI Alignment Forum · 23 sept. 2026
Improve contract search accuracy with auto-generated filters in Amazon Bedrock
AWS · 18 aug. 2026
Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement
arXiv · 5 aug. 2026
Leveraging External Knowledge for Historical Document Restoration via Retrieval-Augmented Large Language Models
arXiv · 27 iul. 2026