aiqbench analysis
The study compares diffusion language models (MDLM, LLaDA) with autoregressive GPT-2 and GPT-Neo on table-to-text generation, scoring reference overlap (BLEU) separately from source grounding (NLI entailment). Adding few-shot demonstrations raised LLaDA's BLEU but lowered entailment, so the two metrics diverge, and the best BLEU score came with no grounding gain. For readers evaluating generative models, the takeaway is that reference-based metrics alone can misrank systems on faithfulness, and the failure modes differ by architecture: autoregressive models drift off the table, while diffusion models more often stay on topic but swap entities.
A Diagnostic Study of Faithfulness in Diffusion Language Models for Table-to-Text Generation
arXiv:2610.29602v1 Announce Type: cross Abstract: Diffusion language models generate by iteratively denoising a masked sequence under bidirectional attention, without the strict left-to-right commitment of autoregressive decoding. Whether this difference affects how reliably they remain grounded in structured inputs is largely unexplored. We study this question in table-to-text generation using ToTTo, where highlighted source cells make factual grounding directly testable. We compare the masked diffusion models MDLM and LLaDA with GPT-2 and GPT-Neo under an identical retrieval-based few-shot protocol, together with a QLoRA-adapted LLaDA, and we score reference similarity and source grounding as separate axes. The two pull apart. Within LLaDA, additional demonstrations raise BLEU from 11.3 to 22.0 while NLI entailment against the source table falls from 0.78 to 0.52; QLoRA reaches the best BLEU of the study, 24.15, and grounds no better for it. Manual annotation shows the failure modes differ in kind: the autoregressive baselines tend to wander off the table, while LLaDA-based models more often stay on topic and swap out individual entities. Reference-based metrics alone are insufficient to assess faithfulness in table-to-text generation.
Abstract:Diffusion language models generate by iteratively denoising a masked sequence under bidirectional attention, without the strict left-to-right commitment of autoregressive decoding. Whether this difference affects how reliably they remain grounded in structured inputs is largely unexplored. We study this question in table-to-text generation using ToTTo, where highlighted source cells make factual grounding directly testable. We compare the masked diffusion models MDLM and LLaDA with GPT-2 and GPT-Neo under an identical retrieval-based few-shot protocol, together with a QLoRA-adapted LLaDA, and we score reference similarity and source grounding as separate axes. The two pull apart. Within LLaDA, additional demonstrations raise BLEU from 11.3 to 22.0 while NLI entailment against the source table falls from 0.78 to 0.52; QLoRA reaches the best BLEU of the study, 24.15, and grounds no better for it. Manual annotation shows the failure modes differ in kind: the autoregressive baselines tend to wander off the table, while LLaDA-based models more often stay on topic and swap out individual entities. Reference-based metrics alone are insufficient to assess faithfulness in table-to-text generation.
9 pages, including references and appendix. Diagnostic study on source faithfulness in table-to-text generation
Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2610.29602 [cs.CL]
(or arXiv:2610.29602v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.29602 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 284 words