analiza aiqbench
This paper introduces a novel correlational training method for morphological neural networks, addressing limitations in gradient-based optimization for layers with sparse Jacobians. By leveraging a Multiplicative Weights Update scheme inspired by learning from experts, the approach achieves significant performance gains and reduced variability across transformer architectures. The work advances fundamental machine learning techniques by offering a more robust alternative to standard back-propagation for specific network topologies.
Correlational Training of Morphological Neural Networks
arXiv:2610.11740v1 Announce Type: new Abstract: Neural networks are typically trained using first-order methods and back-propagation. It is unclear whether this approach is optimal for morphological layers whose weight Jacobians are sparse and whose resulting parameter gradients can be poor. In this work, we propose a novel weight update method for morphological neural networks inspired from the Multiplicative Weights Update (MWU) scheme. We view each morphological perceptron as an instance of the learning from experts' advice problem in logarithmic space, and use a correlation-based reward that favors inputs aligned with the desired output change, regardless of whether a strong gradient signal has reached their weight. We empirically evaluate our approach by training fully connected layers both as stand-alone models and as parts of larger transformer networks. Across nine benchmarks, correlational training yields improvements on eight, by up to 32.84 percentage points, while substantially reducing run-to-run variability.
Abstract:Neural networks are typically trained using first-order methods and back-propagation. It is unclear whether this approach is optimal for morphological layers whose weight Jacobians are sparse and whose resulting parameter gradients can be poor. In this work, we propose a novel weight update method for morphological neural networks inspired from the Multiplicative Weights Update (MWU) scheme. We view each morphological perceptron as an instance of the learning from experts' advice problem in logarithmic space, and use a correlation-based reward that favors inputs aligned with the desired output change, regardless of whether a strong gradient signal has reached their weight. We empirically evaluate our approach by training fully connected layers both as stand-alone models and as parts of larger transformer networks. Across nine benchmarks, correlational training yields improvements on eight, by up to 32.84 percentage points, while substantially reducing run-to-run variability.
Machine Learning (cs.LG)
arXiv:2610.11740 [cs.LG]
(or arXiv:2610.11740v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.11740 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 234 cuvinte
Subiecte conexe
Synthesis Superintelligence: from Semiconductors to Superconductors — Periodic Labs’ Liam Fedus and Ekin Dogus Cubuk
Latent Space · 8 oct. 2026
New AI framework matches scans to standard pathology tests
Quantum Zeitgeist · 3 sept. 2026
Not Deciphering The Voynich Manuscript
LessWrong · 11 oct. 2026
PIRAMID: Progress and Plans
LessWrong · 27 iul. 2026