MLCC: A Congestion Control Technique to Accelerate ML Training

Abstract:We present MLCC, a novel technique to augment today's congestion control algorithms to accelerate DNN training jobs in shared GPU clusters in a fully distributed manner. At the heart of MLCC lies a straightforward principle: DNN training flows should scale their sending rate to shift other flows' communication into their compute periods, achieving interleaving. We show that integrating this principle into today's congestion control protocols is simple (requiring less than 60 lines of code for a given protocol) and enables DNN jobs to interleave within a few training iterations, thereby reducing network contention and improving job completion times. Our testbed demonstrates that MLCC accelerates the average and 99th percentile training iteration times by up to 1.9x and 2.7x respectively. Through extensive packet-level simulations, we observe a 1.35x improvement in training throughput on a 36-node, 288 GPU fat-tree topology.

Anton A. Zabreyko, Sanjoli Narang: Equal Contribution

Networking and Internet Architecture (cs.NI); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)

arXiv:2402.09589 [cs.NI]

(or arXiv:2402.09589v2 [cs.NI] for this version)

https://doi.org/10.48550/arXiv.2402.09589 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.