aiqbench analysis
This technical report introduces LARC, a method enabling frozen models to adapt via low-rank residual connections without full fine-tuning. By decoupling slow task-agnostic factors from fast feedback-driven states, the approach demonstrates significant error reduction in program-selection tasks while analyzing the trade-offs of online updates versus static resets. The study provides rigorous algebraic and empirical validation for maintaining model utility in dynamic environments with minimal parameter overhead.
LARC: Low-Rank Adaptive Residual Connections for Learning in Frozen Models
arXiv:2609.40063v1 Announce Type: cross Abstract: Low-Rank Adaptive Residual Connections (LARC) give a frozen model a compact numerical state that can learn from feedback. The map $h+BAh$ adds a low-rank correction to a hidden representation. A slow state $\rho$ learns starting factors across tasks; a private fast state $\Phi$ copies them, changes with feedback, and resets to the trained initialization. This report specifies an input-side realization of the numerical policy carrier in Memory-Mediated Learning Architecture and examines its factor-space dynamics and learning lifetime. We study a rank-4 input residual with 12,288 trainable parameters on a frozen MiniCPM5-1B-SFT substrate. In a four-candidate program-selection task, two feedback-gradient steps reduce expected query execution error by 24.65 and 36.65 percentage points relative to resetting to the respective trained static and post-adaptation initializations. These development results cover 16 parameter groups and three paired training seeds. A direct support-loss selection rule is much more accurate, reaching 0.78125% error. In a repository-balanced chronological replay of public continuous-integration jobs, retaining online updates raises half-Brier loss from 0.1274 to 0.1808. A fixed follow-up intervention records same-batch non-descent and inconsistent future benefit from shrinking updates. Together, the algebra and measurements distinguish residual capacity, adaptation relative to a starting point, and usefulness on later decisions.
Abstract:Low-Rank Adaptive Residual Connections (LARC) give a frozen model a compact numerical state that can learn from feedback. The map $h+BAh$ adds a low-rank correction to a hidden representation. A slow state $\rho$ learns starting factors across tasks; a private fast state $\Phi$ copies them, changes with feedback, and resets to the trained initialization. This report specifies an input-side realization of the numerical policy carrier in Memory-Mediated Learning Architecture and examines its factor-space dynamics and learning lifetime. We study a rank-4 input residual with 12,288 trainable parameters on a frozen MiniCPM5-1B-SFT substrate. In a four-candidate program-selection task, two feedback-gradient steps reduce expected query execution error by 24.65 and 36.65 percentage points relative to resetting to the respective trained static and post-adaptation initializations. These development results cover 16 parameter groups and three paired training seeds. A direct support-loss selection rule is much more accurate, reaching 0.78125% error. In a repository-balanced chronological replay of public continuous-integration jobs, retaining online updates raises half-Brier loss from 0.1274 to 0.1808. A fixed follow-up intervention records same-batch non-descent and inconsistent future benefit from shrinking updates. Together, the algebra and measurements distinguish residual capacity, adaptation relative to a starting point, and usefulness on later decisions.
19 pages, 6 figures, 15 tables. Technical report of MMLA. The authors contributed equally
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2609.40063 [cs.LG]
(or arXiv:2609.40063v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.40063 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 310 words
Related coverage
Reflections on unlearning and inoculation
LessWrong · Sep 20, 2026
Circuit Hypernetworks for Quantum-Augmented Diffusion Language Models
arXiv · Sep 22, 2026
Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mapping
arXiv · Oct 10, 2026
PIRAMID: Progress and Plans
LessWrong · Jul 27, 2026