Worst-Case Optimal $\sqrt{\epsilon}$-Rate Stability of KL Divergence under Perturbations of Gaussian Distributions

Abstract:We establish a stability bound for the Kullback-Leibler (KL) divergence when its Gaussian reference distribution is perturbed. Existing relaxed triangle inequalities of KL divergence between Gaussian distributions require all three distributions to be Gaussian, which limits their use in applications such as out-of-distribution detection with flow-based generative models. Let $\mathcal{N}_1$ and $\mathcal{N}_2$ be $d$-dimensional Gaussian distributions, and let $P$ be any $d$-dimensional distribution satisfying $\mathbb{E}_P[\|X\|^2]<\infty$ and $\mathrm{KL}(P\Vert \mathcal{N}_1)<\infty$. If $\mathrm{KL}(\mathcal{N}_1\Vert\mathcal{N}_2)\leq\epsilon$ for $0<\epsilon<1/12$, we give an upper bound of $\left|\mathrm{KL}(P\Vert\mathcal{N}_1)-\mathrm{KL}(P\Vert\mathcal{N}_2)\right|$. Under suitable uniform boundedness conditions, the bound is $ \frac{\sqrt{6d\epsilon}}{2}+K\sqrt{\epsilon}+o(\sqrt{\epsilon})$ as $\epsilon\rightarrow 0$ for fixed $d$, where $K$ depends on the parameters of $\mathcal{N}_1$ and $\mathcal{N}_2$, and on $\mathbb{E}_P[\|X\|]$ and $\mathbb{E}_P[\|X\|^2]$. Furthermore, we prove that the $\sqrt{\epsilon}$ dependence is worst-case optimal. The result extends stability guarantees restricted to Gaussian triples to comparisons involving an arbitrary distribution and two nearby Gaussian references. Experiments on synthetic distributions and latent representations from flow-based models support the predicted stability behavior.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2604.11026 [cs.LG]

(or arXiv:2604.11026v4 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2604.11026 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.