aiqbench analysis
This paper introduces DP-Muon, a differentially private optimization algorithm that leverages matrix-orthogonalized momentum to significantly reduce output bias caused by Gaussian noise in nonlinear maps. By deriving exact Gaussian heat identities and proving stationarity bounds under global clipping, the authors demonstrate that their approach outperforms standard Adam baselines on GPT-2 tasks across multiple privacy targets. The work advances the theoretical understanding of adaptive history conditioning in private optimization, offering a more efficient path to training large models with strict privacy guarantees.
DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum
arXiv:2605.12994v2 Announce Type: replace Abstract: We study differentially private optimization with matrix-orthogonalized momentum. DP-Muon uses conventional global per-example clipping and one Gaussian gradient release per step; matrix updates and auxiliary updates are post-processing. Our main contribution concerns the additional mean distortion created when fresh Gaussian noise passes through a nonlinear matrix map. Conditioning on the actual adaptive history immediately before the current noise yields an exact Gaussian heat identity. For a smooth Newton-Schulz map, first-order DP-MuonBC reduces this conditional output bias from second to fourth order in the fresh noise scale, and an arbitrary-order extension has bias of order $2K+2$. We prove matrix-block stationarity bounds under global clipping, retain finite-step orthogonalization error explicitly, and give an exact criterion for improvement of the resulting upper bound. A separate inequality exposes the effect of auxiliary Adam updates. GPT-2 experiments on E2E at four privacy targets favor the reported Muon configurations over Adam baselines in test NLL.
Abstract:We study differentially private optimization with matrix-orthogonalized momentum. DP-Muon uses conventional global per-example clipping and one Gaussian gradient release per step; matrix updates and auxiliary updates are post-processing. Our main contribution concerns the additional mean distortion created when fresh Gaussian noise passes through a nonlinear matrix map. Conditioning on the actual adaptive history immediately before the current noise yields an exact Gaussian heat identity. For a smooth Newton-Schulz map, first-order DP-MuonBC reduces this conditional output bias from second to fourth order in the fresh noise scale, and an arbitrary-order extension has bias of order $2K+2$. We prove matrix-block stationarity bounds under global clipping, retain finite-step orthogonalization error explicitly, and give an exact criterion for improvement of the resulting upper bound. A separate inequality exposes the effect of auxiliary Adam updates. GPT-2 experiments on E2E at four privacy targets favor the reported Muon configurations over Adam baselines in test NLL.
Machine Learning (cs.LG)
arXiv:2605.12994 [cs.LG]
(or arXiv:2605.12994v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2605.12994 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 241 words
Related coverage
Inner Momentum for Differentially Private Muon
arXiv · Oct 5, 2026
Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron
NVIDIA · Apr 22, 2026
Nonsmooth Optimization via Orthogonalized Momentum
arXiv · Sep 16, 2026
A Physical Response-and-Memory Model for Muon Optimization
arXiv · Aug 25, 2026