Mode Collapse of Mean-Field Variational Inference

arXiv:2510.17063v2 Announce Type: replace-cross Abstract: Mean-field variational inference (MFVI) is a widely used method for approximating high-dimensional probability distributions by product measures. It has been empirically observed that MFVI optimizers often suffer from \emph{mode collapse}. For example, when the target measure $\pi$ is a mixture $\pi = w P_0 + (1 - w) P_1$, the MFVI optimizer tends to place most of its mass near a single component of the mixture. This work provides the first \emph{theoretical} explanation of mode collapse in MFVI. Starting from two-component mixtures, we introduce the notion of \emph{$\varepsilon$-separateness} to quantify the separation of the mixture components. We derive explicit bounds on the mass that any MFVI optimizer assigns to each component, when $P_0$ and $P_1$ are $\varepsilon$-separated for sufficiently small $\varepsilon$. We further extend this analysis to mixture target distributions with multiple components. Our results reveal that the occurrence of mode collapse crucially depends on the relative position of the components. To address this issue, we propose \emph{rotational variational inference (RoVI)}, which augments MFVI with a rotation matrix. The numerical studies support our theoretical findings and illustrate improved mode coverage with RoVI in Bayesian mixture posteriors, including a real-data example.

Abstract:Mean-field variational inference (MFVI) is a widely used method for approximating high-dimensional probability distributions by product measures. It has been empirically observed that MFVI optimizers often suffer from \emph{mode collapse}. For example, when the target measure $\pi$ is a mixture $\pi = w P_0 + (1 - w) P_1$, the MFVI optimizer tends to place most of its mass near a single component of the mixture. This work provides the first \emph{theoretical} explanation of mode collapse in MFVI. Starting from two-component mixtures, we introduce the notion of \emph{$\varepsilon$-separateness} to quantify the separation of the mixture components. We derive explicit bounds on the mass that any MFVI optimizer assigns to each component, when $P_0$ and $P_1$ are $\varepsilon$-separated for sufficiently small $\varepsilon$. We further extend this analysis to mixture target distributions with multiple components. Our results reveal that the occurrence of mode collapse crucially depends on the relative position of the components. To address this issue, we propose \emph{rotational variational inference (RoVI)}, which augments MFVI with a rotation matrix. The numerical studies support our theoretical findings and illustrate improved mode coverage with RoVI in Bayesian mixture posteriors, including a real-data example.

Machine Learning (stat.ML); Machine Learning (cs.LG)

arXiv:2510.17063 [stat.ML]

(or arXiv:2510.17063v2 [stat.ML] for this version)

https://doi.org/10.48550/arXiv.2510.17063 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Mode Collapse of Mean-Field Variational Inference

Subiecte conexe