analiza aiqbench
Palimpsa introduces Bayesian metaplasticity to self-attention models, framing in-context learning as a continual learning problem that resolves the stability-plasticity dilemma. By grounding attention state plasticity in importance priors, the architecture effectively expands memory capacity and mitigates interference in long sequences. Theoretically unifying various gated linear attention models, including positioning Mamba2 as a forgetting-dominated special case, this approach offers a robust framework for improving associative recall and commonsense reasoning.
Learning to Remember, Learn, and Forget in Attention-Based Models
arXiv:2602.09075v4 Announce Type: replace Abstract: In-Context Learning (ICL) in transformers acts as an online associative memory and is believed to underpin their high performance on complex sequence processing tasks. However, in gated linear attention models, this memory has a fixed capacity and is prone to interference, especially for long sequences. We propose Palimpsa, a self-attention model that views ICL as a continual learning problem that must address a stability-plasticity dilemma. Palimpsa uses Bayesian metaplasticity, where the plasticity of each attention state is tied to an importance state grounded by a prior distribution that captures accumulated knowledge. We demonstrate that various gated linear attention models emerge as specific architecture choices and posterior approximations, and that Mamba2 is a special case of Palimpsa where forgetting dominates. This theoretical link enables the transformation of any non-metaplastic model into a metaplastic one, significantly expanding its memory capacity. Our experiments show that Palimpsa consistently outperforms baselines on the Multi-Query Associative Recall (MQAR) benchmark and on Commonsense Reasoning tasks.
Abstract:In-Context Learning (ICL) in transformers acts as an online associative memory and is believed to underpin their high performance on complex sequence processing tasks. However, in gated linear attention models, this memory has a fixed capacity and is prone to interference, especially for long sequences. We propose Palimpsa, a self-attention model that views ICL as a continual learning problem that must address a stability-plasticity dilemma. Palimpsa uses Bayesian metaplasticity, where the plasticity of each attention state is tied to an importance state grounded by a prior distribution that captures accumulated knowledge. We demonstrate that various gated linear attention models emerge as specific architecture choices and posterior approximations, and that Mamba2 is a special case of Palimpsa where forgetting dominates. This theoretical link enables the transformation of any non-metaplastic model into a metaplastic one, significantly expanding its memory capacity. Our experiments show that Palimpsa consistently outperforms baselines on the Multi-Query Associative Recall (MQAR) benchmark and on Commonsense Reasoning tasks.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2602.09075 [cs.LG]
(or arXiv:2602.09075v4 [cs.LG] for this version)
Text integral sub licența deschisă a publicației · 172 cuvinte
Subiecte conexe
OpenRouter: from Seed to Stripe — with OpenRouter’s Alex Atallah & AMP’s Anjney Midha
Latent Space · 25 sept. 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · 12 iun. 2026
It’s about ethics in journalism, with Ben Smith
The Verge · 17 aug. 2026
The art and science of hyperparameter optimization on Amazon Nova Forge
AWS · 2 iun. 2026