FILO: A Unified Fisher-Geometric Framework for Low-Rank Adaptation

Abstract:Low-Rank Adaptation (LoRA) reparameterizes a weight matrix $W$ as a product of two low-rank factors, but the Jacobian $J_G$ of the generator mapping the factors to the weight matrix is rank-deficient, so the factor-space preconditioner $J_G^* F_t J_G$ induced by any $W$-space preconditioner $F_t$ is singular, and consequently the standard chain rule cannot be uniquely inverted to map a preconditioned $W$-space direction back to a factor-space update. Therefore, we introduce FILO, a unified Fisher-geometric framework for low-rank adaptation that enables factor updates without directly inverting this singular operator and casts existing LoRA optimizers by their factor-space inversion strategy and $W$-space preconditioner $F_t$. Existing methods occupy four families along these axes: factor-space adaptive updates, block-diagonal surrogates for $J_G^* J_G$, Frobenius-residual pseudoinverse methods, and Riemannian manifold methods. Within this design space, a gradient-statistics-aware $F_t$ paired with a closed-form factor-space solve at $O((m+n)r)$ memory remains underexplored. We instantiate FILO as FILO-Adafactor by adopting the Adafactor diagonal Kronecker preconditioner $H_t$ on $W$ and selecting from the resulting factor-space solution family the element minimizing an $H_t$-weighted imbalance between the two factor contributions. By construction, the induced weight update is the closest LoRA approximation to the preconditioned $W$-space direction under the $H_t$-weighted norm. Across GPT-2 on E2E, Mistral-7B and Qwen2-7B on GLUE, ARC, and GSM8K, and diffusion-model personalization, FILO-Adafactor and its momentum variant are competitive with or improve over a representative set of LoRA optimizers while keeping peak GPU memory at the LoRA optimizer level.

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)

arXiv:2605.08734 [cs.LG]

(or arXiv:2605.08734v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.08734 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.