aiqbench analysis
This arXiv paper (cs.LG) proposes an analytic method for propagating a Gaussian's mean and covariance layer by layer through deep residual networks, rather than relying on Monte Carlo sampling. The reported KL-divergence gains are large on random networks, and the variational Bayes results are a meaningful but narrower benchmark. Readers working on uncertainty quantification or Bayesian neural networks should watch whether the approach holds up at realistic scale and on tasks beyond the synthetic and small-model settings tested.
Exact Gaussian Moment Matching for Residual Networks: a Second-Order Method
arXiv:2601.22307v3 Announce Type: replace Abstract: We study the problem of propagating the mean and covariance of a general multivariate Gaussian distribution through a deep (residual) neural network using layer-by-layer moment matching. We close a longstanding gap by deriving exact moment matching for the probit, GeLU, ReLU (as a limit of GeLU), Heaviside (as a limit of probit), and sine activation functions; for both feedforward and generalized residual layers. On large-variance random networks, median KL-divergence improvements over common baselines range from tens to thousands of times. On a variational Bayes neural network, we show that our method improves upon a state-of-the-art deterministic inference method by an order of magnitude in KL divergence from Monte Carlo ground truth. We also give a smooth-distance error bound showing that, under regularity assumptions, moment matching removes the leading low-variance errors and propagates higher-order local accuracy through the layers of a network.
Abstract:We study the problem of propagating the mean and covariance of a general multivariate Gaussian distribution through a deep (residual) neural network using layer-by-layer moment matching. We close a longstanding gap by deriving exact moment matching for the probit, GeLU, ReLU (as a limit of GeLU), Heaviside (as a limit of probit), and sine activation functions; for both feedforward and generalized residual layers. On large-variance random networks, median KL-divergence improvements over common baselines range from tens to thousands of times. On a variational Bayes neural network, we show that our method improves upon a state-of-the-art deterministic inference method by an order of magnitude in KL divergence from Monte Carlo ground truth. We also give a smooth-distance error bound showing that, under regularity assumptions, moment matching removes the leading low-variance errors and propagates higher-order local accuracy through the layers of a network.
Machine Learning (cs.LG); Numerical Analysis (math.NA)
arXiv:2601.22307 [cs.LG]
(or arXiv:2601.22307v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2601.22307 Focus to learn more arXiv-issued DOI via DataCite
demo/__pycache__/bayesian-network.cpython-312.pyc
demo/__pycache__/bayesian-network.cpython-313.pyc
demo/__pycache__/california_housing.cpython-313.pyc
demo/__pycache__/deep_sine_convergence.cpython-312.pyc
demo/__pycache__/deep_sine_convergence.cpython-313.pyc
demo/__pycache__/stochastic.cpython-313.pyc
demo/__pycache__/test_case.cpython-313.pyc
demo/__pycache__/test_case_contest.cpython-313.pyc
demo/assets/test_case_contest_entries.csv
demo/bayesian-network.py
demo/bayesian.py
demo/deep_sine_convergence.py
demo/runtime_benchmark_wide_deep_residual.ipynb
demo/stochastic.py
demo/test_case.py
demo/test_case_summary.py
demo/trainable_uq.py
neural-uq-supplement.pdf
neural_uncertainty_propagation/__pycache__/__init__.cpython-313.pyc
neural_uncertainty_propagation/__pycache__/activation.cpython-313.pyc
neural_uncertainty_propagation/__pycache__/binorm.cpython-312.pyc
neural_uncertainty_propagation/__pycache__/binorm.cpython-313.pyc
neural_uncertainty_propagation/__pycache__/kalman_diagnostics.cpython-313.pyc
neural_uncertainty_propagation/__pycache__/network.cpython-312.pyc
neural_uncertainty_propagation/__pycache__/network.cpython-313.pyc
neural_uncertainty_propagation/__pycache__/neural_kalman.cpython-313.pyc
neural_uncertainty_propagation/__pycache__/normal.cpython-312.pyc
neural_uncertainty_propagation/__pycache__/normal.cpython-313.pyc
neural_uncertainty_propagation/__pycache__/probit_network.cpython-312.pyc
neural_uncertainty_propagation/__pycache__/probit_network.cpython-313.pyc
neural_uncertainty_propagation/__pycache__/random_matrix.cpython-312.pyc
neural_uncertainty_propagation/__pycache__/random_matrix.cpython-313.pyc
neural_uncertainty_propagation/__pycache__/unscented.cpython-312.pyc
neural_uncertainty_propagation/__pycache__/unscented.cpython-313.pyc
neural_uncertainty_propagation/activation.py
neural_uncertainty_propagation/kalman_diagnostics.py
neural_uncertainty_propagation/network.py
neural_uncertainty_propagation/neural_kalman.py
neural_uncertainty_propagation/normal.py
neural_uncertainty_propagation/random_matrix.py
neural_uncertainty_propagation/unscented.py
(36 additional files not shown) You must enabled JavaScript to view entire file list.
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 291 words