aiqbench analysis
This research introduces DiHAL, a training-free framework that identifies optimal interfaces for integrating continuous diffusion into language models by analyzing representation geometry rather than relying solely on layer depth. By replacing transformer prefixes with conditional diffusion at geometrically selected points, the method achieves performance comparable to full fine-tuning while offering a novel design variable for hybrid model architectures. The findings provide a practical pathway for scaling continuous diffusion without prohibitive computational costs or extensive retraining.
Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement
arXiv:2605.14368v2 Announce Type: replace Abstract: Continuous diffusion language models require choosing a representation space in which denoising operates, yet it remains unclear which representations are most compatible with diffusion. We study a basic design question: where inside a pretrained language model should continuous diffusion operate? We formulate this as a hidden-state interface-selection problem and hypothesize that diffusion-friendly interfaces can be identified from representation geometry. We operationalize this hypothesis using three training-free geometric proxies: local compactness, global stiffness, and effective rank. Across two 8B-scale backbones, the resulting geometry score predicts fixed-budget diffusion bridgeability beyond the dominant effect of layer depth. We then instantiate DiHAL, a Locate-and-Replace framework that replaces the transformer prefix below a selected interface with conditional diffusion while retaining the pretrained suffix and LM head. Under full training, geometry-selected interfaces remain close to validation-loss oracles and outperform worst-layer controls, while the diffusion bridge outperforms a parameter-matched deterministic replacement under matched diagnostic conditions. These results suggest that the representation space in which diffusion operates should itself be treated as a first-class design variable for continuous diffusion in language models.
Title:Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement
Abstract:Continuous diffusion language models require choosing a representation space in which denoising operates, yet it remains unclear which representations are most compatible with diffusion. We study a basic design question: where inside a pretrained language model should continuous diffusion operate? We formulate this as a hidden-state interface-selection problem and hypothesize that diffusion-friendly interfaces can be identified from representation geometry. We operationalize this hypothesis using three training-free geometric proxies: local compactness, global stiffness, and effective rank. Across two 8B-scale backbones, the resulting geometry score predicts fixed-budget diffusion bridgeability beyond the dominant effect of layer depth. We then instantiate DiHAL, a Locate-and-Replace framework that replaces the transformer prefix below a selected interface with conditional diffusion while retaining the pretrained suffix and LM head. Under full training, geometry-selected interfaces remain close to validation-loss oracles and outperform worst-layer controls, while the diffusion bridge outperforms a parameter-matched deterministic replacement under matched diagnostic conditions. These results suggest that the representation space in which diffusion operates should itself be treated as a first-class design variable for continuous diffusion in language models.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2605.14368 [cs.CL]
(or arXiv:2605.14368v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.14368 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 280 words
Related coverage
Direct Preference Optimization Beyond Chatbots
Hugging Face · Jun 3, 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · Oct 8, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
HorizonNav: Geometry-Guided Video Diffusion for Anticipatory 3D Robot Navigation
arXiv · Oct 12, 2026