analiza aiqbench
This research introduces a novel learned optimizer that dynamically recombines gradient history to reduce memory costs while maintaining high performance across diverse model architectures. By generalizing zero-shot to unseen tasks with minimal computational overhead, the method offers a significant efficiency improvement over standard optimizers like Adam for both language and vision models.
Lightweight and Versatile Learned Optimization by Recombination of Gradient History
arXiv:2610.09604v1 Announce Type: new Abstract: This paper presents a lightweight and versatile learned optimizer that dynamically recombines gradient history, represented as averages over disjoint time spans. The optimizer reduces the prediction space to one scalar coefficient per gradient average, shared by multiple parameters. Progressively averaging older gradients minimizes memory cost of long history, while keeping their contributions independently accessible. A 37k-parameter network trained in 0.87 GPU-hours generalizes zero-shot to unseen tasks, lowering validation loss by 9.1% and 0.4% on BERT-Tiny and GPT-Tiny, and improving test accuracy over Adam by 3.5 %p on a Vision Transformer and by 2.7 %p on average across nine graph models, with FLOPs overhead as low as 0.3%.
Abstract:This paper presents a lightweight and versatile learned optimizer that dynamically recombines gradient history, represented as averages over disjoint time spans. The optimizer reduces the prediction space to one scalar coefficient per gradient average, shared by multiple parameters. Progressively averaging older gradients minimizes memory cost of long history, while keeping their contributions independently accessible. A 37k-parameter network trained in 0.87 GPU-hours generalizes zero-shot to unseen tasks, lowering validation loss by 9.1% and 0.4% on BERT-Tiny and GPT-Tiny, and improving test accuracy over Adam by 3.5 %p on a Vision Transformer and by 2.7 %p on average across nine graph models, with FLOPs overhead as low as 0.3%.
Machine Learning (cs.LG)
arXiv:2610.09604 [cs.LG]
(or arXiv:2610.09604v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.09604 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 201 cuvinte
Subiecte conexe
Game Design Creativity evals
LessWrong · 12 oct. 2026
GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model
Meta Engineering · 3 aug. 2026
Online Learning with Gradient-Variation Interval Regret
arXiv · 3 iun. 2026
FOGO: Forgetting-aware Orthogonalization Optimizer
arXiv · 10 iun. 2026