analiza aiqbench
ParaBlock introduces a parallel communication-computation architecture to mitigate latency in federated block coordinate descent for large language models. Theoretically matching standard convergence rates while empirically boosting efficiency, the method specifically targets resource-constrained clients by optimizing parameter transmission during fine-tuning. This approach offers a practical pathway to scaling federated learning on LLMs without sacrificing model performance.
ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
arXiv:2511.19959v2 Announce Type: replace Abstract: Federated learning (FL) has been extensively studied as a privacy-preserving training paradigm. Recently, federated block coordinate descent scheme has become a popular option in training large-scale models, as it allows clients to train only a subset of the model locally instead of the entire model. However, in the era of large language models (LLMs), even a single block can contain a significant number of parameters, posing substantial communication latency, particularly for resource-constrained clients. To address this challenge in federated training/fine-tuning LLMs, we propose ParaBlock, a novel approach that establishes two parallel threads for communication and computation to enhance communication efficiency. We theoretically prove that the proposed ParaBlock achieves the same convergence rate as the standard federated block coordinate descent methods. Empirical evaluations on fine-tuning LLMs on general instruction following and mathematical reasoning confirm that ParaBlock not only maintains strong performance but also significantly improves communication efficiency.
Abstract:Federated learning (FL) has been extensively studied as a privacy-preserving training paradigm. Recently, federated block coordinate descent scheme has become a popular option in training large-scale models, as it allows clients to train only a subset of the model locally instead of the entire model. However, in the era of large language models (LLMs), even a single block can contain a significant number of parameters, posing substantial communication latency, particularly for resource-constrained clients. To address this challenge in federated training/fine-tuning LLMs, we propose ParaBlock, a novel approach that establishes two parallel threads for communication and computation to enhance communication efficiency. We theoretically prove that the proposed ParaBlock achieves the same convergence rate as the standard federated block coordinate descent methods. Empirical evaluations on fine-tuning LLMs on general instruction following and mathematical reasoning confirm that ParaBlock not only maintains strong performance but also significantly improves communication efficiency.
Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)
arXiv:2511.19959 [cs.LG]
(or arXiv:2511.19959v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2511.19959 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 244 cuvinte
Subiecte conexe
News
Quantum Computing Report · 25 iul. 2026
News
Quantum Computing Report · 7 iun. 2026
Building Blocks for Foundation Model Training and Inference on AWS
Hugging Face · 11 mai 2026
Ampere: Communication-Efficient and High-Accuracy Split Federated Learning
arXiv · 31 aug. 2026