analiza aiqbench
DiScoFormer addresses a fundamental trade-off in generative modeling by unifying the generalization of classical kernel methods with the precision of neural score models into a single, train-once architecture. Theoretically grounded in the ability of self-attention to recover normalized KDE, the model offers a versatile plug-in oracle for density estimation, score computation, and solving Fokker-Planck-type PDEs without retraining. This approach effectively mitigates the curse of dimensionality while maintaining high fidelity across varying distributions and sample sizes.
DiScoFormer: Plug-In Density and Score Estimation with Transformers
arXiv:2511.05924v4 Announce Type: replace Abstract: Estimating probability density and its score from samples remains a core problem in generative modeling, Bayesian inference, and kinetic theory. Existing methods are bifurcated: classical kernel density estimators (KDE) generalize across distributions but suffer from the curse of dimensionality, while modern neural score models achieve high precision but require retraining for every target distribution. We introduce DiScoFormer (Density and Score Transformer), a ``train-once, infer-anywhere" equivariant Transformer that maps i.i.d. samples to both density values and score vectors, generalizing across distributions and sample sizes. Analytically, we prove that self-attention can recover normalized KDE, establishing it as a functional generalization of kernel methods; empirically, individual attention heads learn multi-scale, kernel-like behaviors. The model converges faster and achieves higher precision than KDE for density estimation, and provides a high-fidelity plug-in score oracle for score-debiased KDE, Fisher information computation, and Fokker-Planck-type PDEs.
Abstract:Estimating probability density and its score from samples remains a core problem in generative modeling, Bayesian inference, and kinetic theory. Existing methods are bifurcated: classical kernel density estimators (KDE) generalize across distributions but suffer from the curse of dimensionality, while modern neural score models achieve high precision but require retraining for every target distribution. We introduce DiScoFormer (Density and Score Transformer), a ``train-once, infer-anywhere" equivariant Transformer that maps i.i.d. samples to both density values and score vectors, generalizing across distributions and sample sizes. Analytically, we prove that self-attention can recover normalized KDE, establishing it as a functional generalization of kernel methods; empirically, individual attention heads learn multi-scale, kernel-like behaviors. The model converges faster and achieves higher precision than KDE for density estimation, and provides a high-fidelity plug-in score oracle for score-debiased KDE, Fisher information computation, and Fokker-Planck-type PDEs.
Machine Learning (cs.LG)
arXiv:2511.05924 [cs.LG]
(or arXiv:2511.05924v4 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 149 cuvinte
Subiecte conexe
DiScoFormer: One transformer for density and score, across distributions
Hugging Face · 29 iun. 2026
News Archive 2026
Quantum Computing Report · 25 iul. 2026
Causal Density Functions
arXiv · 28 iul. 2026
Deconfusing Corrigibility
AI Alignment Forum · 17 oct. 2026