analiza aiqbench
This research paper proposes a unified theory defining forgetting in machine learning as a loss of self-consistency within predictive distributions. By demonstrating that exact Bayesian inference enables adaptation without memory degradation, the study offers new pathways to improve information retention across classification, generative modeling, and reinforcement learning tasks. These findings are critical for advancing general-purpose algorithms capable of lifelong learning without catastrophic interference from prior knowledge.
Forgetting is Everywhere
arXiv:2511.04666v4 Announce Type: replace Abstract: A fundamental challenge in developing general learning algorithms is their tendency to forget past knowledge as they adapt to new data. Addressing this problem requires a principled understanding of forgetting. Yet, despite decades of study, no unified definition has emerged that offers insight into the underlying dynamics of learning. We propose an algorithm- and task-agnostic theory that characterises forgetting as a lack of self-consistency in a learner's predictive distribution, manifesting as a loss of predictive information. Our theory naturally yields a general measure of an algorithm's propensity to forget, proves that exact Bayesian inference allows for adaptation without forgetting, and provides a tautological explanation for why generative models forget when trained on their own synthetic outputs. To validate these claims, we design a comprehensive set of experiments that span classification, regression, generative modelling, and reinforcement learning. We demonstrate that forgetting is present across all deep learning settings and plays a significant role in determining learning efficiency. Together, these results establish a principled understanding of forgetting and lay the foundation for analysing and improving the information retention capabilities of general learning algorithms.
Abstract:A fundamental challenge in developing general learning algorithms is their tendency to forget past knowledge as they adapt to new data. Addressing this problem requires a principled understanding of forgetting. Yet, despite decades of study, no unified definition has emerged that offers insight into the underlying dynamics of learning. We propose an algorithm- and task-agnostic theory that characterises forgetting as a lack of self-consistency in a learner's predictive distribution, manifesting as a loss of predictive information. Our theory naturally yields a general measure of an algorithm's propensity to forget, proves that exact Bayesian inference allows for adaptation without forgetting, and provides a tautological explanation for why generative models forget when trained on their own synthetic outputs. To validate these claims, we design a comprehensive set of experiments that span classification, regression, generative modelling, and reinforcement learning. We demonstrate that forgetting is present across all deep learning settings and plays a significant role in determining learning efficiency. Together, these results establish a principled understanding of forgetting and lay the foundation for analysing and improving the information retention capabilities of general learning algorithms.
Machine Learning (cs.LG); Machine Learning (stat.ML)
arXiv:2511.04666 [cs.LG]
(or arXiv:2511.04666v4 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 195 cuvinte
Subiecte conexe
It’s about ethics in journalism, with Ben Smith
The Verge · 17 aug. 2026
The current bottleneck is political will, not research
AI Alignment Forum · 11 iul. 2026
Benn Jordan on Flock, privacy, and the new surveillance backlash
The Verge · 15 oct. 2026
OpenRouter: from Seed to Stripe — with OpenRouter’s Alex Atallah & AMP’s Anjney Midha
Latent Space · 25 sept. 2026