aiqbench analysis
Bayesian-LoRA addresses the critical issue of miscalibration in large language models by reformulating deterministic Low-Rank Adaptation into a probabilistic framework inspired by Sparse Gaussian Processes. This approach significantly improves uncertainty quantification and calibration on small datasets with minimal overhead, offering a robust solution for reliable model deployment where confidence scores must reflect true prediction certainty.
Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
arXiv:2601.21003v3 Announce Type: replace Abstract: Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes. We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning benchmarks. With only approximately 0.42M additional parameters and ${\approx}1.2{\times}$ training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B, achieving up to 84% ECE reduction and 76% NLL reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.
Abstract:Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes. We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning benchmarks. With only approximately 0.42M additional parameters and ${\approx}1.2{\times}$ training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B, achieving up to 84% ECE reduction and 76% NLL reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.
Artificial Intelligence (cs.AI)
arXiv:2601.21003 [cs.AI]
(or arXiv:2601.21003v3 [cs.AI] for this version)
Full text under the publisher's open licence · 150 words
Related coverage
Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation
arXiv · Sep 1, 2026
ReLoRA: Knowledge-Reusing Adaptation for Fast Rollout of Evolving LLM Services
arXiv · Jun 3, 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
Reflections on unlearning and inoculation
LessWrong · Sep 20, 2026