Separation capacity of linear reservoirs with random connectivity matrix

Abstract:A natural hypothesis for the success of reservoir computing in generic tasks is the ability of the untrained reservoir to map distinct input time series to separable reservoir states, a property we term separation capacity. In this work, we develop a rigorous mathematical framework for analysing the separation capacity of random linear reservoirs. We show that the expected separation induced by a random reservoir is completely characterised by the spectral properties of a positive semi-definite matrix naturally associated with the connectivity matrix, which we call the generalised matrix of moments. Taking Gaussian connectivity matrices as an example, we investigate how separation depends on the reservoir dimension $N$, the scaling of the connectivity matrix, and structural assumptions such as symmetry. In the symmetric case, we show that, although the classical scaling $N^{-1/2}$ yields the most balanced separation for large reservoirs, the quality of separation inevitably deteriorates as the length of the input time series increases. In contrast, for reservoirs with independent and identically distributed connectivity entries, we prove that the classical scaling $N^{-1/2}$ is asymptotically optimal from the perspective of separation and derive quantitative bounds describing the evolution of separation with the time horizon. Finally, numerical experiments suggest a strong practical connection between balanced separation profiles and downstream learning performance. Beyond providing a theoretical justification for several common reservoir design choices, our results introduce separation capacity as a tractable and informative framework for the analysis of random reservoirs.

Machine Learning (stat.ML); Machine Learning (cs.LG); Probability (math.PR)

arXiv:2404.17429 [stat.ML]

(or arXiv:2404.17429v4 [stat.ML] for this version)

https://doi.org/10.48550/arXiv.2404.17429 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.