aiqbench analysis
This arXiv paper is squarely about AI: it proposes PACS, a supervised fine-tuning method for diffusion language models that uses the model's own predictability to choose masking and attention-visible input length. The reported gains (up to 12.84 points over vanilla SFT on reasoning and code benchmarks) are benchmark results from a single preprint, so they merit replication before being treated as established. For practitioners, the key takeaway is that random masking during SFT can misalign training with how diffusion models actually decode, which is a useful design consideration for anyone building non-autoregressive LLMs.
Match the Canvas, Not Just the Answer: Predictability-Aware SFT for Diffusion Language Models
arXiv:2610.18001v1 Announce Type: cross Abstract: Diffusion language models (DLMs) enable flexible parallel generation by resolving tokens according to the model's evolving predictability rather than a fixed left-to-right order. This flexibility, however, introduces training choices absent from standard autoregressive teacher forcing: which reference tokens are visible when a target is supervised and how much generation space is allocated. Standard supervised fine-tuning (SFT) typically makes these choices through random masking, independently of inference. Our analysis shows how this training-inference mismatch can misdirect supervision. Random masking can conceal prediction errors or penalize uncertainty that decoding would otherwise resolve, while the attention-visible input length can change which response formulations the model favors, even when the underlying solution is unchanged. Motivated by these findings, we propose Predictability-Aware Canvas Selection and Supervision (PACS), an SFT method. PACS uses the model's own predictability to select each example's attention-visible input length and supervises reference tokens immediately before their confidence-guided revelation. Across reasoning and code-generation benchmarks, PACS consistently achieves the highest mean accuracy among the compared SFT methods, with gains of up to 12.84 percentage points over vanilla SFT on individual benchmarks. Code will be released upon publication.
Authors:Miao Li, Hanyang Jiang, Hengyu Fu, Baihe Huang, Yating Liu, Yuhang Cai, Changkun Guan, Zhongzhu Chen, Jiecheng Lu, Michael A. Boateng, Pascal Van Hentenryck
Abstract:Diffusion language models (DLMs) enable flexible parallel generation by resolving tokens according to the model's evolving predictability rather than a fixed left-to-right order. This flexibility, however, introduces training choices absent from standard autoregressive teacher forcing: which reference tokens are visible when a target is supervised and how much generation space is allocated. Standard supervised fine-tuning (SFT) typically makes these choices through random masking, independently of inference. Our analysis shows how this training-inference mismatch can misdirect supervision. Random masking can conceal prediction errors or penalize uncertainty that decoding would otherwise resolve, while the attention-visible input length can change which response formulations the model favors, even when the underlying solution is unchanged. Motivated by these findings, we propose Predictability-Aware Canvas Selection and Supervision (PACS), an SFT method. PACS uses the model's own predictability to select each example's attention-visible input length and supervises reference tokens immediately before their confidence-guided revelation. Across reasoning and code-generation benchmarks, PACS consistently achieves the highest mean accuracy among the compared SFT methods, with gains of up to 12.84 percentage points over vanilla SFT on individual benchmarks. Code will be released upon publication.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2610.18001 [cs.LG]
(or arXiv:2610.18001v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.18001 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 233 words
Related coverage
Inception in DiffusionGemma - Jailbreaking a Diffusion Language Model by Pinning Tokens Anywhere on the Canvas
LessWrong · Jul 23, 2026
DiffusionGemma: Google’s Diffusion-Based Open Model for Faster Text Generation
Analytics Vidhya · Jun 11, 2026
[AINews] TypeSafe/Jev at >$100M ARR, $7.5B valuation 3 weeks after launch
Latent Space · Oct 10, 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · May 19, 2026