aiqbench analysis
This research introduces CONEC-LoRA, a parameter-efficient fine-tuning method designed to mitigate catastrophic forgetting in domain incremental learning by consolidating shared and task-specific knowledge. By integrating stochastic classifiers and an auxiliary network for optimal LoRA selection, the approach demonstrates significant accuracy improvements over existing linear or prototype-based methods on standard benchmarks.
Continual Knowledge Consolidation LORA for Domain Incremental Learning
arXiv:2510.16077v2 Announce Type: replace-cross Abstract: Domain Incremental Learning (DIL) is a sub-branch of continual learning that aims to address the never-ending arrival of new domains without catastrophic forgetting. Despite the advent of parameter-efficient fine-tuning (PEFT) approaches, prior works create task-specific LoRAs that overlook shared knowledge across tasks. Inaccurate selection of task-specific LoRAs during inference leads to significant drops in accuracy, while existing works rely on linear or prototype-based classifiers, which have suboptimal generalization powers. Our paper proposes continual knowledge consolidation low-rank adaptation (CONEC-LoRA) addressing the DIL problems. CONEC-LoRA is developed from consolidations between task-shared LORA to extract common knowledge and task-specific LORA to incorporate domain-specific knowledge. Unlike existing approaches, CONEC-LoRA integrates the concept of a stochastic classifier whose parameters are sampled from a distribution, thus enhancing the likelihood of correct classifications. Last but not least, an auxiliary network is deployed to optimally predict the task-specific LoRAs for inference and implements the concept of a different-depth network structure in which every layer is connected with a local classifier to leverage intermediate representations. This module integrates the ball-generator loss and transformation module to address the synthetic sample bias problem. Our rigorous experiments demonstrate the advantage of CONEC-LoRA over prior art on 4 popular benchmark problems with over 5% margins.
Abstract:Domain Incremental Learning (DIL) is a sub-branch of continual learning that aims to address the never-ending arrival of new domains without catastrophic forgetting. Despite the advent of parameter-efficient fine-tuning (PEFT) approaches, prior works create task-specific LoRAs that overlook shared knowledge across tasks. Inaccurate selection of task-specific LoRAs during inference leads to significant drops in accuracy, while existing works rely on linear or prototype-based classifiers, which have suboptimal generalization powers. Our paper proposes continual knowledge consolidation low-rank adaptation (CONEC-LoRA) addressing the DIL problems. CONEC-LoRA is developed from consolidations between task-shared LORA to extract common knowledge and task-specific LORA to incorporate domain-specific knowledge. Unlike existing approaches, CONEC-LoRA integrates the concept of a stochastic classifier whose parameters are sampled from a distribution, thus enhancing the likelihood of correct classifications. Last but not least, an auxiliary network is deployed to optimally predict the task-specific LoRAs for inference and implements the concept of a different-depth network structure in which every layer is connected with a local classifier to leverage intermediate representations. This module integrates the ball-generator loss and transformation module to address the synthetic sample bias problem. Our rigorous experiments demonstrate the advantage of CONEC-LoRA over prior art on 4 popular benchmark problems with over 5% margins.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2510.16077 [cs.LG]
(or arXiv:2510.16077v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2510.16077 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 297 words
Related coverage
RegCL: Compact Continual SAM Adaptation for Visual Grounding in Multi-Sensorial Media
arXiv · Aug 31, 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · Jun 12, 2026
The art and science of hyperparameter optimization on Amazon Nova Forge
AWS · Jun 2, 2026
Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning
arXiv · Aug 5, 2026