analiza aiqbench
This research challenges the prevailing trend toward asynchronous optimization by theoretically proving that synchronous SGD methods are nearly optimal in many heterogeneous distributed learning scenarios. The study demonstrates that robust synchronous variants can handle random delays and partial worker failures without sacrificing convergence rates, suggesting they may be sufficient for a wide range of modern AI training regimes.
Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods
arXiv:2602.03802v2 Announce Type: replace-cross Abstract: Modern distributed optimization methods mostly rely on traditional synchronous approaches, despite substantial recent progress in asynchronous optimization. We revisit Synchronous SGD and its robust variant, called $m$-Synchronous SGD, which can be interpreted as a method with a backup-worker strategy, and theoretically show that they are nearly optimal in many heterogeneous computation and communication scenarios, which is somewhat unexpected. We analyze the synchronous methods under random times and adversarial partial participation of workers, and prove that their time complexities are optimal in many practical regimes, up to logarithmic factors. While synchronous methods are not universal solutions and there exist tasks where asynchronous methods may be necessary, we show theoretically that they are sufficient for many modern heterogeneous computation and communication scenarios.
Abstract:Modern distributed optimization methods mostly rely on traditional synchronous approaches, despite substantial recent progress in asynchronous optimization. We revisit Synchronous SGD and its robust variant, called $m$-Synchronous SGD, which can be interpreted as a method with a backup-worker strategy, and theoretically show that they are nearly optimal in many heterogeneous computation and communication scenarios, which is somewhat unexpected. We analyze the synchronous methods under random times and adversarial partial participation of workers, and prove that their time complexities are optimal in many practical regimes, up to logarithmic factors. While synchronous methods are not universal solutions and there exist tasks where asynchronous methods may be necessary, we show theoretically that they are sufficient for many modern heterogeneous computation and communication scenarios.
Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Numerical Analysis (math.NA); Optimization and Control (math.OC)
arXiv:2602.03802 [cs.DC]
(or arXiv:2602.03802v2 [cs.DC] for this version)
Text integral sub licența deschisă a publicației · 144 cuvinte