aiqbench analysis
This paper argues that JEPA-style self-supervised models should inject structural bias into the view-to-view predictive coupling rather than forcing one-view embeddings toward an isotropic Gaussian. Its HamJEPA instantiation, which uses a learned Hamiltonian leapfrog map on phase-space states, reports gains over SIGReg on CIFAR-100 and ImageNet-100. The results are small-scale benchmarks with short training runs, and the authors note that matched non-symplectic and non-reversible controls perform similarly, so the Hamiltonian structure itself is not shown to be necessary. Readers should treat the theory as a useful framing and the empirical gains as preliminary until tested at larger scale.
Beyond Isotropy in JEPAs: Hamiltonian Geometry and Symplectic Prediction
arXiv:2605.20107v2 Announce Type: replace-cross Abstract: JEPAs often regularize one-view embeddings toward an isotropic Gaussian, implicitly baking Euclidean symmetry into the representation. We show that this is not merely a benign default. For a known structured downstream geometry $H\succ0$, the minimax and maximum-entropy covariance under a Hamiltonian energy budget is $(c/d)H^{-1}$, and Euclidean isotropy incurs a closed-form price of isotropy. More importantly, when the downstream geometry is unknown, no geometry-independent fixed marginal target is canonical: every fixed covariance shape can be maximally misaligned for some structured geometry. We further show that even oracle one-view marginals do not identify the JEPA view-to-view predictive coupling. Within continuous linear phase-space flows, stationarity of the lifted oracle law, momentum reversal, and fixed-unit kinematics uniquely select Hamiltonian dynamics. These results suggest that the structural bias in JEPAs should enter the cross-view coupling rather than a fixed encoder marginal. We instantiate this principle with \textbf{HamJEPA}, which encodes each view as a phase-space state $(q,p)$ and predicts view-to-view transitions with a learned Hamiltonian leapfrog map, while non-isotropic scale and spectral floors discourage collapse. In a deliberately headless token protocol, HamJEPA improves over SIGReg on CIFAR-100 by $+4.89$ kNN@20 and $+3.52$ linear-probe points at 30 epochs, and, over two training seeds at 80 epochs, by $+7.24$ kNN@20 and $+9.67$ linear-probe points relative to the full 2048-dimensional SIGReg representation. Matched non-symplectic and non-reversible controls retain the gain, so the experiments do not identify either property as individually necessary. On ImageNet-100, HamJEPA-$q$ improves by $+4.82$ kNN@20 and $+7.52$ linear-probe points at 45 epochs.
Abstract:JEPAs often regularize one-view embeddings toward an isotropic Gaussian, implicitly baking Euclidean symmetry into the representation. We show that this is not merely a benign default. For a known structured downstream geometry $H\succ0$, the minimax and maximum-entropy covariance under a Hamiltonian energy budget is $(c/d)H^{-1}$, and Euclidean isotropy incurs a closed-form price of isotropy. More importantly, when the downstream geometry is unknown, no geometry-independent fixed marginal target is canonical: every fixed covariance shape can be maximally misaligned for some structured geometry. We further show that even oracle one-view marginals do not identify the JEPA view-to-view predictive coupling. Within continuous linear phase-space flows, stationarity of the lifted oracle law, momentum reversal, and fixed-unit kinematics uniquely select Hamiltonian dynamics. These results suggest that the structural bias in JEPAs should enter the cross-view coupling rather than a fixed encoder marginal. We instantiate this principle with \textbf{HamJEPA}, which encodes each view as a phase-space state $(q,p)$ and predicts view-to-view transitions with a learned Hamiltonian leapfrog map, while non-isotropic scale and spectral floors discourage collapse. In a deliberately headless token protocol, HamJEPA improves over SIGReg on CIFAR-100 by $+4.89$ kNN@20 and $+3.52$ linear-probe points at 30 epochs, and, over two training seeds at 80 epochs, by $+7.24$ kNN@20 and $+9.67$ linear-probe points relative to the full 2048-dimensional SIGReg representation. Matched non-symplectic and non-reversible controls retain the gain, so the experiments do not identify either property as individually necessary. On ImageNet-100, HamJEPA-$q$ improves by $+4.82$ kNN@20 and $+7.52$ linear-probe points at 45 epochs.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2605.20107 [cs.LG]
(or arXiv:2605.20107v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.20107 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 342 words
Related coverage
CoSynFlow: Conformal Symplectic Neural Flows for Cross-System Prediction of Dissipative Hamiltonian Dynamics
arXiv · Aug 4, 2026
Symplectic Representation of Legendre Dynamics
arXiv · Aug 3, 2026
Area Scaling of Dynamical Degrees of Freedom in Regularised Scalar Field Theory
arXiv · Aug 25, 2026
LeHami: Symplectic Latent Dynamics for World Models
arXiv · Oct 15, 2026