Learning Mamba as a Continual Learner: Meta-Learning Selective State Space Models for Continual Learning

Abstract:Continual learning (CL) learns from a non-stationary data stream without storing or re-training on all seen samples. Meta-continual learning (MCL) casts CL as sequence prediction and meta-learns the continual learner itself as a sequence model, with Transformers as natural choices. However, despite decent performance, a Transformer learner relies on a linearly growing key-value cache to store all past representations, conflicting with CL's objective of not storing all seen samples. Sequence models with a constant-size state, e.g., linear-attention models and state-space models (SSMs), match CL's requirement by definition; however, with past samples compressed and mixed in the bounded state, such learners are harder to meta-learn, and earlier kernel-based ones performed poorly on MCL. We thus aim to obtain effective continual learners with constant-size states and make them work for MCL. By formulating the selective SSM for MCL, we propose MambaCL, with Mamba as the main model in practice. We meta-train the learner with a proposed selectivity regularization, which supervises the implicit associations within the compressed state and applies across constant-state models. Furthermore, we conduct a systematic empirical study of how Mamba and other constant-state models behave across various MCL scenarios, covering effectiveness, generalization, and model designs. The results highlight promising performance and strong generalization, demonstrating the potential of constant-state learners for efficient continual learning and adaptation.

Machine Learning (cs.LG)

arXiv:2412.00776 [cs.LG]

(or arXiv:2412.00776v5 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2412.00776 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.