On associative neural networks for sparse patterns with huge capacities

Abstract:Generalized Hopfield models with higher-order or exponential interaction terms are known to have substantially larger storage capacities than the classical quadratic model. On the other hand, associative memories for sparse patterns, such as the Willshaw and Amari models, already exhibit enhanced storage capacities in the sparse regime. In this paper we combine these two mechanisms. We introduce higher-order versions of sparse associative memory models and study their storage capacities in the sense of fixed-pattern stability. For the Amari and Willshaw models with fixed interaction order $n$, we obtain storage scales of order $\frac{N^n}{(\log N)^n}$. When the interaction order grows logarithmically with the number of neurons, the resulting storage scale becomes super-polynomial. We also study higher-order interactions in the block-structured Gripon--Berrou architecture, where the natural storage scale is of order $c^n$. Our results show that the capacity increase caused by higher-order interactions persists in the sparse setting, while the precise storage scale depends on the underlying architecture.

Probability (math.PR); Machine Learning (cs.LG)

arXiv:2603.26217 [math.PR]

(or arXiv:2603.26217v2 [math.PR] for this version)

https://doi.org/10.48550/arXiv.2603.26217 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.