Continual Knowledge Consolidation LORA for Domain Incremental Learning

arXiv:2510.16077v2 Announce Type: replace-cross Abstract: Domain Incremental Learning (DIL) is a sub-branch of continual learning that aims to address the never-ending arrival of new domains without catastrophic forgetting. Despite the advent of parameter-efficient fine-tuning (PEFT) approaches, prior works create task-specific LoRAs that overlook shared knowledge across tasks. Inaccurate selection of task-specific LoRAs during inference leads to significant drops in accuracy, while existing works rely on linear or prototype-based classifiers, which have suboptimal generalization powers. Our paper proposes continual knowledge consolidation low-rank adaptation (CONEC-LoRA) addressing the DIL problems. CONEC-LoRA is developed from consolidations between task-shared LORA to extract common knowledge and task-specific LORA to incorporate domain-specific knowledge. Unlike existing approaches, CONEC-LoRA integrates the concept of a stochastic classifier whose parameters are sampled from a distribution, thus enhancing the likelihood of correct classifications. Last but not least, an auxiliary network is deployed to optimally predict the task-specific LoRAs for inference and implements the concept of a different-depth network structure in which every layer is connected with a local classifier to leverage intermediate representations. This module integrates the ball-generator loss and transformation module to address the synthetic sample bias problem. Our rigorous experiments demonstrate the advantage of CONEC-LoRA over prior art on 4 popular benchmark problems with over 5% margins.

Abstract:Domain Incremental Learning (DIL) is a sub-branch of continual learning that aims to address the never-ending arrival of new domains without catastrophic forgetting. Despite the advent of parameter-efficient fine-tuning (PEFT) approaches, prior works create task-specific LoRAs that overlook shared knowledge across tasks. Inaccurate selection of task-specific LoRAs during inference leads to significant drops in accuracy, while existing works rely on linear or prototype-based classifiers, which have suboptimal generalization powers. Our paper proposes continual knowledge consolidation low-rank adaptation (CONEC-LoRA) addressing the DIL problems. CONEC-LoRA is developed from consolidations between task-shared LORA to extract common knowledge and task-specific LORA to incorporate domain-specific knowledge. Unlike existing approaches, CONEC-LoRA integrates the concept of a stochastic classifier whose parameters are sampled from a distribution, thus enhancing the likelihood of correct classifications. Last but not least, an auxiliary network is deployed to optimally predict the task-specific LoRAs for inference and implements the concept of a different-depth network structure in which every layer is connected with a local classifier to leverage intermediate representations. This module integrates the ball-generator loss and transformation module to address the synthetic sample bias problem. Our rigorous experiments demonstrate the advantage of CONEC-LoRA over prior art on 4 popular benchmark problems with over 5% margins.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

arXiv:2510.16077 [cs.LG]

(or arXiv:2510.16077v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2510.16077 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Continual Knowledge Consolidation LORA for Domain Incremental Learning

Subiecte conexe