aiqbench analysis
This research introduces CFLoRA, a novel federated learning scheme that resolves structural mismatches in LoRA factorization to enable exact aggregation without freezing parameters or expanding communication ranks. By partitioning latent channels into complementary sets, the method achieves superior training efficiency and convergence rates while supporting heterogeneous client resources. The approach represents a significant advancement for deploying collaborative large language model fine-tuning across privacy-sensitive environments.
CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
arXiv:2609.33147v1 Announce Type: new Abstract: Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present \texttt{CFLoRA}, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates \texttt{CFLoRA} achieves $\mathcal{O}(1/\sqrt{T})$ convergence rate of the \textit{original} LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that \texttt{CFLoRA} achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.
Abstract:Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present \texttt{CFLoRA}, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates \texttt{CFLoRA} achieves $\mathcal{O}(1/\sqrt{T})$ convergence rate of the \textit{original} LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that \texttt{CFLoRA} achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.
Machine Learning (cs.LG)
arXiv:2609.33147 [cs.LG]
(or arXiv:2609.33147v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.33147 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 265 words
Related coverage
MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning
arXiv · Aug 18, 2026
FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon
arXiv · Sep 9, 2026
SplitLite: Low-Rank Residual Compression for Split Learning
arXiv · Aug 25, 2026
FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation
arXiv · Aug 5, 2026