aiqbench analysis
This comparative study benchmarks four major Federated Learning frameworks—FedML, Flower, Substra, and OpenFL—focusing on scalability metrics across varying client counts. The analysis reveals that while some platforms struggle with resource efficiency or accuracy degradation at scale, OpenFL demonstrates superior stability in CPU/RAM usage and consistent performance growth. These findings offer critical guidance for practitioners selecting distributed training infrastructure where model privacy and system reliability are paramount.
Scalability and Performance Evaluation of Federated Learning Frameworks: A Comparative Analysis
arXiv:2609.15681v1 Announce Type: cross Abstract: This paper presents a systematic examination and experimental comparison of the prominent Federated Learning (FL) frameworks FedML, Flower, Substra, and OpenFL. The frameworks are evaluated experimentally by implementing Federated Learning over a varying number of clients, emphasizing a thorough analysis of scalability and key performance metrics. The study assesses the impact of increasing client counts on total training time, loss and accuracy values, and CPU and RAM usage. Results indicate distinct performance characteristics among the frameworks, with Flower displaying an unusually high loss, FedML achieving a notably low accuracy range of 66% to 79%, and Substra demonstrating good resource efficiency, albeit with an exponential growth in total training time. Notably, OpenFL emerges as the most scalable platform, demonstrating consistent accuracy, loss, and training time across different client counts. OpenFL's stable CPU and RAM underscore its reliability in real-world scenarios. This comprehensive analysis provides valuable insights into the relative performance of FL frameworks, offering good understanding of their capabilities and providing guidance for their effective deployment across diverse user bases.
Abstract:This paper presents a systematic examination and experimental comparison of the prominent Federated Learning (FL) frameworks FedML, Flower, Substra, and OpenFL. The frameworks are evaluated experimentally by implementing Federated Learning over a varying number of clients, emphasizing a thorough analysis of scalability and key performance metrics. The study assesses the impact of increasing client counts on total training time, loss and accuracy values, and CPU and RAM usage. Results indicate distinct performance characteristics among the frameworks, with Flower displaying an unusually high loss, FedML achieving a notably low accuracy range of 66% to 79%, and Substra demonstrating good resource efficiency, albeit with an exponential growth in total training time. Notably, OpenFL emerges as the most scalable platform, demonstrating consistent accuracy, loss, and training time across different client counts. OpenFL's stable CPU and RAM underscore its reliability in real-world scenarios. This comprehensive analysis provides valuable insights into the relative performance of FL frameworks, offering good understanding of their capabilities and providing guidance for their effective deployment across diverse user bases.
Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI)
arXiv:2609.15681 [cs.DC]
(or arXiv:2609.15681v1 [cs.DC] for this version)
https://doi.org/10.48550/arXiv.2609.15681 arXiv-issued DOI via DataCite (pending registration)
Soudan, B., Abbas, S., Kubba, A. et al. Scalability and performance evaluation of federated learning frameworks: a comparative analysis. Int. J. Mach. Learn. & Cyber. 16, 3329-3343 (2025)
https://doi.org/10.1007/s13042-024-02453-4 DOI(s) linking to related resources
Full text under the publisher's open licence · 227 words
