aiqbench analysis
The paper adds motion equivariance to convolutional LSTMs by treating a learned velocity model as a gauge field, so the network's memory moves with the input rather than staying fixed. The reported gains are over a convolutional LSTM and a flow-equivariant baseline at equal parameter counts, on future-frame prediction, motion-only digit classification, and action recognition under camera shake. Readers should weigh the "provably equivariant" claim against benchmarks that are still small-scale, and watch whether the approach transfers to large real-world video models.
Motion Equivariant Convolutional LSTM: When Motion Becomes a Gauge Symmetry
arXiv:2610.28771v1 Announce Type: new Abstract: Objects move, cameras pan, and weather drifts with the wind: much of what changes in sequential data is motion. Motion changes where things are, not what they are, and is therefore a symmetry of the data. Yet standard recurrent networks are not equivariant to it: their memory stays put while the input moves on. Flow equivariant recurrent networks first brought this symmetry into sequence models, for motions of constant velocity, or flows, by enumerating a bank of velocities. We extend this idea in three directions: from simple recurrent networks to convolutional LSTMs, from flows to motions whose velocity changes in time, and from enumerating velocities to estimating them. Treating a sequence as a field over time, we show that motion becomes a gauge symmetry: a global symmetry made local in time. Built on this view, our motion equivariant convolutional LSTM (MELSTM) uses its velocity model as the gauge field and is provably equivariant to every motion. Across future-frame prediction, classification of digits visible only through their motion, and action recognition under camera shake, MELSTM outperforms a convolutional LSTM and its flow equivariant counterpart with the same number of parameters: it is more accurate, generalizes to longer horizons and to unseen motions, and carries a fraction of the flow equivariant model's state.
Abstract:Objects move, cameras pan, and weather drifts with the wind: much of what changes in sequential data is motion. Motion changes where things are, not what they are, and is therefore a symmetry of the data. Yet standard recurrent networks are not equivariant to it: their memory stays put while the input moves on. Flow equivariant recurrent networks first brought this symmetry into sequence models, for motions of constant velocity, or flows, by enumerating a bank of velocities. We extend this idea in three directions: from simple recurrent networks to convolutional LSTMs, from flows to motions whose velocity changes in time, and from enumerating velocities to estimating them. Treating a sequence as a field over time, we show that motion becomes a gauge symmetry: a global symmetry made local in time. Built on this view, our motion equivariant convolutional LSTM (MELSTM) uses its velocity model as the gauge field and is provably equivariant to every motion. Across future-frame prediction, classification of digits visible only through their motion, and action recognition under camera shake, MELSTM outperforms a convolutional LSTM and its flow equivariant counterpart with the same number of parameters: it is more accurate, generalizes to longer horizons and to unseen motions, and carries a fraction of the flow equivariant model's state.
Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
arXiv:2610.28771 [cs.LG]
(or arXiv:2610.28771v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.28771 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 311 words
Related coverage
Surrogate Modeling of 3D Rayleigh-Benard Convection with Equivariant Autoencoders
arXiv · Sep 14, 2026
Sobek: Streaming Equivariant Tensor Product Convolutions
arXiv · Jul 21, 2026
Product Interaction: An Algebraic Formalism for Deep Learning Architectures
arXiv · Oct 17, 2026
Moment kernels: a simple and scalable approach for equivariance to rotations and reflections in deep convolutional networks
arXiv · Aug 3, 2026