aiqbench analysis
CLARE addresses the scalability bottleneck in continual learning by utilizing a sparsity-driven framework to accumulate tasks within a shared adapter space without catastrophic forgetting. This approach significantly improves final accuracy on long task sequences, such as Omnibenchmark-1k, by optimizing only task-critical parameters to reduce inter-task interference. The method offers a practical pathway for deploying large models in dynamic environments where sequential learning is required.
CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework
arXiv:2609.17026v1 Announce Type: new Abstract: Continual learning must balance the learning of new knowledge with the retention of previously learned knowledge to incrementally learn tasks from a data stream without catastrophic forgetting. While leveraging pretrained models has significantly advanced continual learning, existing methods exhibit a scalability bottleneck when trained sequentially on many tasks, suffering from performance degradation due to inter-task interference and loss of plasticity. Inspired by evidence that sparse fine-tuning achieves performance comparable to full fine-tuning, this paper presents a novel sparsity-driven continual learning framework. Our continual learning method, termed CLARE, operates in two stages: it first identifies a sparse, task-critical parameter mask via a sparsity-inducing objective, then performs mask-constrained fine-tuning by only optimizing parameters selected by the mask. This two-stage sparse adapter mechanism enables all tasks to be accumulated within a shared adapter space while reducing destructive interference across tasks. Extensive experiments demonstrate the scalability of CLARE. On the long task-sequence benchmark Omnibenchmark-1k, CLARE outperforms strong baselines in final accuracy by a large margin, e.g, improving EASE by 4.64% and 13.34% after learning 100 tasks, respectively.
Abstract:Continual learning must balance the learning of new knowledge with the retention of previously learned knowledge to incrementally learn tasks from a data stream without catastrophic forgetting. While leveraging pretrained models has significantly advanced continual learning, existing methods exhibit a scalability bottleneck when trained sequentially on many tasks, suffering from performance degradation due to inter-task interference and loss of plasticity. Inspired by evidence that sparse fine-tuning achieves performance comparable to full fine-tuning, this paper presents a novel sparsity-driven continual learning framework. Our continual learning method, termed CLARE, operates in two stages: it first identifies a sparse, task-critical parameter mask via a sparsity-inducing objective, then performs mask-constrained fine-tuning by only optimizing parameters selected by the mask. This two-stage sparse adapter mechanism enables all tasks to be accumulated within a shared adapter space while reducing destructive interference across tasks. Extensive experiments demonstrate the scalability of CLARE. On the long task-sequence benchmark Omnibenchmark-1k, CLARE outperforms strong baselines in final accuracy by a large margin, e.g, improving EASE by 4.64% and 13.34% after learning 100 tasks, respectively.
Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2609.17026 [cs.LG]
(or arXiv:2609.17026v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.17026 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 273 words
Related coverage
Class Incremental Continual Learning with Self-Organizing Maps and Synthetic Replay
arXiv · Aug 31, 2026
SPARS: A Stability-Plasticity-Aware Region Selection Framework for Continual Learning
arXiv · Oct 15, 2026
Unsupervised Continual Learning with Growing Self-Organizing Maps and Synthetic Replay
arXiv · Aug 31, 2026
One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning
arXiv · Sep 1, 2026