aiqbench analysis
This research establishes dynamical isometry as a critical mechanism for maintaining neural network plasticity during continual learning, offering a new regularization scheme and the AdamO optimizer to mitigate catastrophic forgetting. By decoupling isometry constraints from standard gradient updates, the proposed method effectively reactivates dormant ReLU units while preserving expressive nonlinear representations. The approach demonstrates consistent performance gains across supervised and reinforcement learning benchmarks, providing a theoretically grounded solution to a persistent challenge in non-stationary training scenarios.
Preserving Plasticity in Continual Learning via Dynamical Isometry
arXiv:2606.09762v1 Announce Type: new Abstract: Continual training of deep neural networks under non-stationarity often leads to a progressive loss of plasticity, eventually limiting further learning. We relate plasticity to the empirical Neural Tangent Kernel, and identify dynamical isometry (the condition that layer-wise Jacobian singular values remain close to one) as a key mechanism for preserving plasticity in continual learning. We revisit a class of networks that are almost-everywhere isometric while remaining universal Lipschitz function approximators, demonstrating that near-dynamical isometry is compatible with expressive nonlinear representations. For general architectures, we propose an efficient isometry-promoting regularization scheme and identify a novel mechanism by which it can reactivate dormant ReLU units. Building on this, we introduce AdamO, an Adam-style adaptive optimizer that decouples isometry regularization from gradient updates, analogous to AdamW. We further reinterpret prior plasticity-preserving approaches through the lens of dynamical isometry, showing that they target only a partial measure of isometry. Across supervised and reinforcement-learning continual-learning benchmarks designed to induce plasticity loss, our methods consistently match or outperform existing approaches.
Abstract:Continual training of deep neural networks under non-stationarity often leads to a progressive loss of plasticity, eventually limiting further learning. We relate plasticity to the empirical Neural Tangent Kernel, and identify dynamical isometry (the condition that layer-wise Jacobian singular values remain close to one) as a key mechanism for preserving plasticity in continual learning. We revisit a class of networks that are almost-everywhere isometric while remaining universal Lipschitz function approximators, demonstrating that near-dynamical isometry is compatible with expressive nonlinear representations. For general architectures, we propose an efficient isometry-promoting regularization scheme and identify a novel mechanism by which it can reactivate dormant ReLU units. Building on this, we introduce AdamO, an Adam-style adaptive optimizer that decouples isometry regularization from gradient updates, analogous to AdamW. We further reinterpret prior plasticity-preserving approaches through the lens of dynamical isometry, showing that they target only a partial measure of isometry. Across supervised and reinforcement-learning continual-learning benchmarks designed to induce plasticity loss, our methods consistently match or outperform existing approaches.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2606.09762 [cs.LG]
(or arXiv:2606.09762v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.09762 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 261 words
Related coverage
Not All Change Is Damage: Update-Matched Unlearning for Large Language Models
arXiv · Oct 16, 2026
Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
arXiv · Jun 10, 2026
Plasticity of Growing and Elastic Neural Networks in Online Continual Learning
arXiv · Aug 4, 2026
Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
arXiv · Oct 15, 2026