How Temporal Correlations Shape Memory in Linear Recurrent Neural Networks

Abstract:The linear recurrent neural network (LRNN) is a simple model for studying how much memory a network builds up as it trains. For uncorrelated inputs, earlier work found that training itself settles the network between keeping the past and reacting only to the present. Real sequences are correlated, and we solve the learning dynamics exactly for correlated inputs. In the solution, keeping the past carries a cost. The whole effect of correlation lands on that cost. This cost reduces to the earlier one when inputs are uncorrelated and grows once they are positively correlated. Three findings follow. (1) Correlation reshapes the course of learning, not only its end. Memory builds, overshoots, and is partly removed, and the settled network keeps less of the past. (2) Memory switches off at a threshold set by one number, how much each input resembles the one just before it. Neither sequence length nor longer-range correlation moves this threshold. Memory is worth keeping only when the task needs the previous input more than the current input already supplies it through correlation with the past. (3) The best network changes too. Zero error demands a feedthrough, a path that passes the current input straight to the network's output and remembers nothing, and training builds it unprompted when given one spare hidden dimension. Our work turns one property of the input into a prediction of whether a network learns memory and explains why correlated data turns recurrent networks into change detectors.

Machine Learning (cs.LG)

arXiv:2609.00420 [cs.LG]

(or arXiv:2609.00420v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2609.00420 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

derivations/lean/Memreg.lean

derivations/lean/Memreg/Basic.lean

derivations/lean/lake-manifest.json

derivations/lean/lakefile.toml

derivations/verify_formulas.py

derivations/verify_transition.py

experiments/capacity_full.py

experiments/recruit_full.py

experiments/robustness.py

experiments/test_recruitment.py

figures/fig_dynamics_price.py

figures/fig_grid_dynamics.py

figures/fig_recruit_grid.py

figures/grid_reference.json

real_data/build_table.py

real_data/core.py

real_data/invariance.py

real_data/series_record.csv

requirements.txt

wshape/wshape_ar1.py

wshape/wshape_ar2.py

wshape/wshape_arma11.py

wshape/wshape_exch.py

wshape/wshape_exch_verify.py

wshape/wshape_longmem.py

wshape/wshape_longmem_v4.py

wshape/wshape_ma1.py

wshape/wshape_seasonal.py

wshape/wshape_seasonal2.py

wshape/wshape_seasonal3.py

wshape/wshape_sum2ar.py

wshape/wshape_sum2ar_check.py

wshape/wshape_sum2ar_min.py

wshape/wshape_sum2ar_robust.py

(31 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.