aiqbench analysis
LoGRA addresses the critical memory bottleneck in reinforcement learning for large language models by utilizing low-rank gradient sketches to compress training signals. This technique reduces average memory usage by nearly 46% while enabling stable training of massive models on limited hardware, making RL post-training more accessible. The method also incorporates predicted-KL step control to manage update magnitudes and prevent learning disruption.
LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches
arXiv:2610.06647v1 Announce Type: new Abstract: Reinforcement learning (RL) has greatly advanced the capabilities of large language models (LLMs), but its memory demands remain a barrier to broader adoption. We introduce LoGRA, an approach to RL post-training that reduces memory by retaining useful learning signals in low-rank gradient sketches. These compact representations support both model updates and efficient policy synchronization. To prevent overly large updates from disrupting learning, we complement gradient compression with predicted-KL step control, which estimates policy changes before applying each update and adjusts its magnitude accordingly. Across reasoning tasks, LoGRA reduces average training memory by up to 45.7\% without sacrificing performance. It also enables stable training of a 27B-parameter model for over 1,100 steps on a single eight-GPU node, where dense Adam runs out of memory, making previously memory-infeasible RL training practical. Code is available in the \href{https://github.com/skzhang1/labs-molt/tree/logra/examples/scripts/logra}{Molt library}.
Abstract:Reinforcement learning (RL) has greatly advanced the capabilities of large language models (LLMs), but its memory demands remain a barrier to broader adoption. We introduce LoGRA, an approach to RL post-training that reduces memory by retaining useful learning signals in low-rank gradient sketches. These compact representations support both model updates and efficient policy synchronization. To prevent overly large updates from disrupting learning, we complement gradient compression with predicted-KL step control, which estimates policy changes before applying each update and adjusts its magnitude accordingly. Across reasoning tasks, LoGRA reduces average training memory by up to 45.7\% without sacrificing performance. It also enables stable training of a 27B-parameter model for over 1,100 steps on a single eight-GPU node, where dense Adam runs out of memory, making previously memory-infeasible RL training practical. Code is available in the \href{this https URL}{Molt library}.
Computation and Language (cs.CL)
arXiv:2610.06647 [cs.CL]
(or arXiv:2610.06647v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.06647 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 233 words
Related coverage
The art and science of hyperparameter optimization on Amazon Nova Forge
AWS · Jun 2, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026
Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards
arXiv · Oct 14, 2026
One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control
arXiv · Sep 9, 2026