aiqbench analysis
DynaTokens addresses the critical challenge of task interference in continual learning by dynamically generating fine-tuning tokens on demand, effectively reducing forgetting in multimodal video-language models without storing impractical task-specific prompts. The method leverages meta-learning-inspired regularizers and gradient-free routing to maintain robust performance across extended domain-incremental sequences and cross-modal transfers. This advancement is significant for deploying large language models in evolving environments where preserving prior knowledge while adapting to new tasks is essential.
DynaTokens: Controlling Token Dynamics for Continual Video-Language Understanding
arXiv:2603.06662v3 Announce Type: replace-cross Abstract: Continual VideoQA with multimodal LLMs remains challenging because sequential adaptation induces task interference, while storing task-specific prompts becomes impractical as task sequences grow. We introduce DynaTokens, a transformer-based token generator that dynamically produces fine-tuning tokens on demand, enabling task-adaptive prompt updates through shared generation weights. To mitigate forgetting, we introduce meta-learning-inspired regularisers that look ahead to avoid task-specific sharp update directions while anchoring the evolving generator to prior-task behaviours. We theoretically connect this objective to sharpness-aware optimisation, showing how it favours flatter cross-task minima and improves retention. DynaTokens combines gradient-free routing based on robust pretrained token and visual embeddings with lightweight auxiliary multimodal supervision, reducing router drift during continual adaptation. Across standard continual VideoQA benchmarks, DynaTokens achieves higher average accuracy and substantially lower forgetting than strong baselines. It also improves zero-shot generalisation and remains effective in longer domain-incremental sequences with extended task shifts. Finally, we introduce a challenging ImageQA->VideoQA protocol and show that DynaTokens enables robust cross-modal continual transfer.
Abstract:Continual VideoQA with multimodal LLMs remains challenging because sequential adaptation induces task interference, while storing task-specific prompts becomes impractical as task sequences grow. We introduce DynaTokens, a transformer-based token generator that dynamically produces fine-tuning tokens on demand, enabling task-adaptive prompt updates through shared generation weights. To mitigate forgetting, we introduce meta-learning-inspired regularisers that look ahead to avoid task-specific sharp update directions while anchoring the evolving generator to prior-task behaviours. We theoretically connect this objective to sharpness-aware optimisation, showing how it favours flatter cross-task minima and improves retention. DynaTokens combines gradient-free routing based on robust pretrained token and visual embeddings with lightweight auxiliary multimodal supervision, reducing router drift during continual adaptation. Across standard continual VideoQA benchmarks, DynaTokens achieves higher average accuracy and substantially lower forgetting than strong baselines. It also improves zero-shot generalisation and remains effective in longer domain-incremental sequences with extended task shifts. Finally, we introduce a challenging ImageQA->VideoQA protocol and show that DynaTokens enables robust cross-modal continual transfer.
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2603.06662 [cs.CV]
(or arXiv:2603.06662v3 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2603.06662 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 257 words
Related coverage
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · Sep 25, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding
arXiv · Jun 8, 2026
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · Sep 4, 2026