aiqbench analysis
This research introduces CRAM, a novel architecture for Multimodal Continual Instruction Tuning that addresses the critical trade-off between catastrophic forgetting and parameter efficiency in Large Language Models. By combining centroid-guided routing with adaptive Mixture-of-Experts mechanisms, the method dynamically allocates parameters to new tasks while preserving existing capabilities through orthogonality constraints. The approach represents a significant advancement for deploying versatile multimodal models in real-world scenarios requiring continuous learning without retraining from scratch.
CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
arXiv:2606.02502v2 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual Instruction Tuning (MCIT) essential. Existing methods either update all tasks with a shared parameter set or allocate dedicated modules for each new task. Shared updates force heterogeneous tasks to compete, causing forgetting of learned capabilities. Conversely, isolated expansion prevents interference but severely limits parameter efficiency over long task streams. To address this dilemma, we propose CRAM (Centroid-Routing and Adaptive MoE). Specifically, by isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks. To further boost parameter efficiency, we utilize adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters. To ensure stable reuse among tasks, centroid-guided routing recognizes and activates existing experts' capabilities, while an orthogonality penalty confines new updates to task-specific directions, preventing re-learning general capability. Extensive experiments across diverse benchmarks demonstrate its superiority over existing methods. Code is available at https://github.com/LAMDA-CL/EMNLP2026-CRAM.
Abstract:Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual Instruction Tuning (MCIT) essential. Existing methods either update all tasks with a shared parameter set or allocate dedicated modules for each new task. Shared updates force heterogeneous tasks to compete, causing forgetting of learned capabilities. Conversely, isolated expansion prevents interference but severely limits parameter efficiency over long task streams. To address this dilemma, we propose CRAM (Centroid-Routing and Adaptive MoE). Specifically, by isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks. To further boost parameter efficiency, we utilize adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters. To ensure stable reuse among tasks, centroid-guided routing recognizes and activates existing experts' capabilities, while an orthogonality penalty confines new updates to task-specific directions, preventing re-learning general capability. Extensive experiments across diverse benchmarks demonstrate its superiority over existing methods. Code is available at this https URL.
Accepted to EMNLP 2026 (Main Conference). Code is available at [this URL](this https URL)
Computation and Language (cs.CL)
arXiv:2606.02502 [cs.CL]
(or arXiv:2606.02502v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.02502 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 283 words
Related coverage
ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning
arXiv · Jun 5, 2026
CART: Compatible Routing for Multimodal Continual Instruction Tuning
arXiv · Oct 17, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026
SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning
arXiv · Aug 27, 2026