analiza aiqbench
This research addresses the critical coupling between LoRA rank selection and uplink payload constraints in wireless federated learning environments. By introducing a two-timescale design that separates offline rank optimization from online sparsification, the proposed SOFT and TSFA methods enable dynamic adaptation to fading channels without requiring full matrix computations. The work advances the efficiency of model fine-tuning over bandwidth-limited networks by leveraging near-orthogonal updates to guide payload allocation.
Two Stage Wireless Federated LoRA Fine-Tuning with Sparsified Orthogonal Updates
arXiv:2505.00333v3 Announce Type: replace Abstract: Federated fine-tuning with low-rank adaptation (LoRA) communicates only two low-rank matrices instead of the full model, but existing methods typically fix the LoRA rank in advance as a manually tuned hyperparameter. In wireless networks, however, the rank determines both adaptation capacity and uplink payload, while the deliverable payload varies with the fading channel. To address this coupling, we formulate wireless federated LoRA fine-tuning as a two-timescale design that separates the \emph{offline-optimized rank}, i.e., the rank of the shared LoRA structure selected before training from statistical channel information, from the per-iteration sparsification and bandwidth decisions adapted to instantaneous CSI. For per-iteration adaptation, we propose sparsified orthogonal fine-tuning (\textbf{SOFT}), which promotes near-orthogonality among rank components so that the product of the corresponding column and row norms approximates each component's singular value. This SVD-free score guides component-wise payload allocation and within-component entry selection without forming the full matrix product. We further derive a convergence bound linking the two timescales and develop a two-stage federated algorithm (\textbf{TSFA}) that selects the offline-optimized rank offline and jointly optimizes sparsification and bandwidth online via Lyapunov optimization.
Abstract:Federated fine-tuning with low-rank adaptation (LoRA) communicates only two low-rank matrices instead of the full model, but existing methods typically fix the LoRA rank in advance as a manually tuned hyperparameter. In wireless networks, however, the rank determines both adaptation capacity and uplink payload, while the deliverable payload varies with the fading channel. To address this coupling, we formulate wireless federated LoRA fine-tuning as a two-timescale design that separates the \emph{offline-optimized rank}, i.e., the rank of the shared LoRA structure selected before training from statistical channel information, from the per-iteration sparsification and bandwidth decisions adapted to instantaneous CSI. For per-iteration adaptation, we propose sparsified orthogonal fine-tuning (\textbf{SOFT}), which promotes near-orthogonality among rank components so that the product of the corresponding column and row norms approximates each component's singular value. This SVD-free score guides component-wise payload allocation and within-component entry selection without forming the full matrix product. We further derive a convergence bound linking the two timescales and develop a two-stage federated algorithm (\textbf{TSFA}) that selects the offline-optimized rank offline and jointly optimizes sparsification and bandwidth online via Lyapunov optimization.
Machine Learning (cs.LG); Signal Processing (eess.SP)
arXiv:2505.00333 [cs.LG]
(or arXiv:2505.00333v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2505.00333 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 274 cuvinte
Subiecte conexe
FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks
arXiv · 16 sept. 2026
[AINews] not much happened today
Latent Space · 3 oct. 2026
The art and science of hyperparameter optimization on Amazon Nova Forge
AWS · 2 iun. 2026
SPECTRA: Band-Routed Embedding and Stage-Wise LoRA for Cross-Sensor Fine-Tuning of Geospatial Foundation Models
arXiv · 5 aug. 2026