analiza aiqbench
The article primarily addresses theoretical limitations in discrete diffusion models and on-policy distillation for reasoning systems, focusing on mathematical dependencies between tokens and sampling distributions. These topics are central to advancing machine learning theory and model training methodologies rather than discussing consumer applications or unrelated tech sectors. TechPulse take: This research identifies fundamental constraints where per-position distributions fail to capture token dependencies in discrete diffusion, potentially limiting the accuracy of generated sequences like text or images. Additionally, it highlights unresolved questions regarding when on-policy distillation signals improve versus degrade reasoning model performance, signaling a need for deeper investigation into supervision contexts.
Limits of Confidence in Diffusion
Discrete diffusion, including remasking and uniform-state samplers, generate a sequence by writing multiple token positions per step, drawing each from a per-position distribution and choosing which positions to write from those same distributions. For domains of general interest (pixels, phonemes, or words) there are inherent dependencies between tokens. We show that a step matches the training distribution only when the positions it writes are conditionally independent given the tokens already fixed, that no product of per-position distributions can match a dependent group, and that…
Discrete diffusion, including remasking and uniform-state samplers, generate a sequence by writing multiple token positions per step, drawing each from a per-position distribution and choosing which positions to write from those same distributions. For domains of general interest (pixels, phonemes, or words) there are inherent dependencies between tokens. We show that a step matches the training distribution only when the positions it writes are conditionally independent given the tokens already fixed, that no product of per-position distributions can match a dependent group, and that per-position distributions do not determine whether a group is dependent: two joint distributions can have identical per-position marginals while differing in which combinations of values occur. On ScanAndAdd, a synthetic task whose joint distribution is available in closed form, we verify that every group of two or more undetermined positions a confidence ranking writes is dependent, and measure the generated distribution to be 29× the sampling-noise floor total variation while per-sample metrics are 1.0.
Related readings and updates.
On-policy distillation offers dense, per-token supervision for training reasoning models; however, it remains unclear under which conditions this signal is beneficial and under which it is detrimental. Which teacher model should be used, and in the case of self-distillation, which specific context should serve as the supervisory signal? Does the optimal choice vary from one token to the next? At present, addressing these questions typically…
Transformers have gained increasing popularity in a wide range of applications, including Natural Language Processing (NLP), Computer Vision and Speech Recognition, because of their powerful representational capacity. However, harnessing this representational capacity effectively requires a large amount of data, strong regularization, or both, to mitigate overfitting. Recently, the power of the Transformer has been unlocked by self-supervised…
Text integral sub licența deschisă a publicației · 284 cuvinte
Subiecte conexe
Problems of Proliferation
LessWrong · 29 sept. 2026
Against Export Controls (and China Threat Models)
LessWrong · 24 sept. 2026
Google AI Releases DiffusionGemma, a 26B MoE Open Model Using Text Diffusion for Up to 4x Faster Generation
MarkTechPost · 10 iun. 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · 29 sept. 2026