aiqbench analysis
This research introduces GRIL, a novel architecture for linear recurrent networks that enables in-context learning via gradient descent-like updates within a single forward pass. By integrating multiplicative readouts and sliding-window self-attention, the model effectively bridges standard sequence modeling with supervised product exposure required for dynamic task adaptation. The approach demonstrates practical viability on both synthetic benchmarks and language modeling tasks like Long Range Arena.
Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks
arXiv:2410.11687v3 Announce Type: replace Abstract: Linear recurrent networks (LRNNs) offer linear-time sequence modeling, but standard recurrent updates do not directly expose the supervised products needed for in-context gradient descent. We propose a sufficient constructive inductive bias for LRNNs: equip a diagonal recurrent state with multiplicative readout and a short sliding-window cross-product self-attention update. The resulting architecture, Gradient-based Recurrent In-context Learner (GRIL), can implement minibatch gradient descent on a task-specific linear predictor during a single forward pass. The same design extends to multi-step updates and cross-entropy classification, with a limited MLP-based extension to non-linear regression. Empirically, trained GRILs recover the behavior and parameters predicted by the construction on synthetic ICL tasks, and the same architectural bias yields useful performance on Long Range Arena and language modelling. These results present windowed cross-product self-attention as a practical, testable inductive bias for LRNNs that learn in context through gradient-descent-like updates.
Abstract:Linear recurrent networks (LRNNs) offer linear-time sequence modeling, but standard recurrent updates do not directly expose the supervised products needed for in-context gradient descent. We propose a sufficient constructive inductive bias for LRNNs: equip a diagonal recurrent state with multiplicative readout and a short sliding-window cross-product self-attention update. The resulting architecture, Gradient-based Recurrent In-context Learner (GRIL), can implement minibatch gradient descent on a task-specific linear predictor during a single forward pass. The same design extends to multi-step updates and cross-entropy classification, with a limited MLP-based extension to non-linear regression. Empirically, trained GRILs recover the behavior and parameters predicted by the construction on synthetic ICL tasks, and the same architectural bias yields useful performance on Long Range Arena and language modelling. These results present windowed cross-product self-attention as a practical, testable inductive bias for LRNNs that learn in context through gradient-descent-like updates.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)
arXiv:2410.11687 [cs.LG]
(or arXiv:2410.11687v3 [cs.LG] for this version)
Full text under the publisher's open licence · 160 words
Related coverage
Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks
arXiv · Oct 8, 2026
Pre-Training Hybrid LLMs from Scratch in Primus on AMD Instinct™ GPUs
AMD · Oct 8, 2026
Gradient descent dynamics for deep equilibrium models
arXiv · Oct 6, 2026
Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications
arXiv · Jun 9, 2026