aiqbench analysis
This paper provides a critical theoretical analysis of how covariance matrix assumptions in variational inference systematically distort the signal-to-noise ratio, directly impacting model selection and generalization bounds. It demonstrates that standard tractability shortcuts like diagonal or low-rank plus diagonal approximations can lead to underfitting, while specific low-rank structures may cause overfitting in Bayesian linear regression. These findings offer essential guidance for researchers scaling Bayesian models, highlighting the trade-off between computational feasibility and accurate hyperparameter learning.
Occam's Razor is Only as Sharp as Your ELBO
arXiv:2604.25984v2 Announce Type: replace-cross Abstract: The marginal likelihood, also known as the evidence, is regarded as a mathematical embodiment of Occam's razor, enabling hyperparameter selection that avoids overfitting. The evidence lower bound (ELBO) objective from variational inference has also been used for similar purposes. In this paper, we define a signal-to-noise ratio (SNR) for Bayesian linear regression in terms of key hyperparameters controlling signal and noise variance. We then study how ELBO-based hyperparameter learning can under or overestimate SNR depending on the assumed covariance matrix structure in a Gaussian approximate posterior. We prove that assuming a diagonal or low-rank plus diagonal covariance matrix underestimates the SNR, while a low-rank covariance matrix overestimates the SNR. Underestimating the SNR leads to higher than optimal mean squared error on the training set (underfitting), while overestimating SNR leads to lower error (overfitting). Bayesian practitioners hoping to scale to large models should be cautious about how covariance assumptions needed for tractability may impact model selection.
Abstract:The marginal likelihood, also known as the evidence, is regarded as a mathematical embodiment of Occam's razor, enabling hyperparameter selection that avoids overfitting. The evidence lower bound (ELBO) objective from variational inference has also been used for similar purposes. In this paper, we define a signal-to-noise ratio (SNR) for Bayesian linear regression in terms of key hyperparameters controlling signal and noise variance. We then study how ELBO-based hyperparameter learning can under or overestimate SNR depending on the assumed covariance matrix structure in a Gaussian approximate posterior. We prove that assuming a diagonal or low-rank plus diagonal covariance matrix underestimates the SNR, while a low-rank covariance matrix overestimates the SNR. Underestimating the SNR leads to higher than optimal mean squared error on the training set (underfitting), while overestimating SNR leads to lower error (overfitting). Bayesian practitioners hoping to scale to large models should be cautious about how covariance assumptions needed for tractability may impact model selection.
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2604.25984 [stat.ML]
(or arXiv:2604.25984v2 [stat.ML] for this version)
https://doi.org/10.48550/arXiv.2604.25984 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 250 words
Related coverage
Prediction Under Imperfect Compression: A Theory of Approximate MDL
arXiv · Jun 5, 2026
Do Language Models Follow Occam's Razor? An Evaluation of Parsimony in Inductive and Abductive Reasoning
arXiv · Aug 28, 2026
Why don’t machine learning research agents overfit?
Amazon Science · Sep 10, 2026
Statistical learning theory and Occam's razor: Regularization
arXiv · Aug 6, 2026