Beyond Square Roots: A Memory-Efficient Explicit Factorization for Multi-Epoch Private Learning

Abstract:Correlated-noise mechanisms are among the most promising approaches for improving the utility of differentially private model training, but rigorous guarantees require explicit, analyzable factorizations, and practical deployment requires memory efficiency. Recent works have developed banded inverse factorizations, which address both requirements by exploiting a banded structure in the correlation matrix. The bandwidth controls the size of the noise buffer used to correlate noise across iterations, and thus governs the tradeoff between utility and memory cost. At the two ends of this tradeoff, DP-$\lambda$CGD achieves high memory efficiency by using only a one-step noise buffer, but this limits its utility gains, while the banded inverse square root (BISR) factorization exploits larger correlation windows and is asymptotically optimal for large bandwidths but performs poorly at low bandwidths. To bridge this gap, we introduce $\gamma$-BIFR, a unified generalization of both factorizations. In the low-memory, low-bandwidth regime, $\gamma$-BIFR improves RMSE, amplified RMSE, and private training performance, while yielding tighter theoretical guarantees for multi-participation error in multi-epoch training.

Machine Learning (cs.LG)

arXiv:2605.18379 [cs.LG]

(or arXiv:2605.18379v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.18379 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.