aiqbench analysis
This research proposes a novel optimization technique using Fisher information to adapt step sizes in stochastic gradient ascent for Dirichlet process mixtures. The method aims to accelerate Bayesian nonparametric inference without compromising the performance typically achieved by closed-form coordinate ascent solutions. By integrating momentum and adaptive steps, this approach facilitates faster training on large-scale datasets while remaining compatible with deep convolutional features.
Fisher Information based Stochastic Gradient Ascent for Online Learning of Dirichlet Process Mixture and Theory
arXiv:2412.08951v3 Announce Type: replace Abstract: Scalable algorithms of posterior approximation allow Bayesian nonparametrics such as Dirichlet process mixture to scale up to larger dataset at fractional cost. Recent algorithms, notably the stochastic variational inference performs local learning from minibatch. The main problem with stochastic variational inference is that it relies on closed form solution. Stochastic gradient ascent is a modern approach to machine learning and is widely deployed in the training of deep neural networks. In this work, we explore using stochastic gradient ascent as a fast algorithm for the posterior approximation of Dirichlet process mixture. However, stochastic gradient ascent alone is not optimal for learning. In order to achieve both speed and performance, we turn our focus to stepsize optimization in stochastic gradient ascent. As as intermediate approach, we first optimize stepsize using the momentum method. Finally, we introduce Fisher information to allow adaptive stepsize in our posterior approximation. In the experiments, we justify that our approach using stochastic gradient ascent do not sacrifice performance for speed when compared to closed form coordinate ascent learning on these datasets. Lastly, our approach is also compatible with deep ConvNet features as well as scalable to large class datasets such as Caltech256 and SUN397.
Abstract:Scalable algorithms of posterior approximation allow Bayesian nonparametrics such as Dirichlet process mixture to scale up to larger dataset at fractional cost. Recent algorithms, notably the stochastic variational inference performs local learning from minibatch. The main problem with stochastic variational inference is that it relies on closed form solution. Stochastic gradient ascent is a modern approach to machine learning and is widely deployed in the training of deep neural networks. In this work, we explore using stochastic gradient ascent as a fast algorithm for the posterior approximation of Dirichlet process mixture. However, stochastic gradient ascent alone is not optimal for learning. In order to achieve both speed and performance, we turn our focus to stepsize optimization in stochastic gradient ascent. As as intermediate approach, we first optimize stepsize using the momentum method. Finally, we introduce Fisher information to allow adaptive stepsize in our posterior approximation. In the experiments, we justify that our approach using stochastic gradient ascent do not sacrifice performance for speed when compared to closed form coordinate ascent learning on these datasets. Lastly, our approach is also compatible with deep ConvNet features as well as scalable to large class datasets such as Caltech256 and SUN397.
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2412.08951 [cs.LG]
(or arXiv:2412.08951v3 [cs.LG] for this version)
Full text under the publisher's open licence · 211 words
Related coverage
Exploding variance of means of exponentials: least-squares to the rescue – Machine Learning Research Blog
francisbach.com · Sep 27, 2026
Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning
arXiv · Jul 23, 2026
Performance Evaluation of Ising and QUBO Variable Encodings in Boltzmann Machine Learning
arXiv · Oct 5, 2026
Variance Reduction for Non-Log-Concave Sampling with Applications to Inverse Problems
arXiv · Jun 16, 2026