analiza aiqbench
EfficientXpert introduces a novel framework that co-adapts low-rank updates with sparse support to optimize domain-specialized large language models on resource-constrained hardware. By integrating a ForeSight Mask for weight scoring and a Partial Brain Surgeon for adapter realignment, the method achieves near-dense performance at 40% sparsity while managing training overhead effectively. This approach directly addresses the trade-off between adaptation cost and model capacity retention, offering practical guidance for deploying efficient experts in production environments.
EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning
arXiv:2511.19935v3 Announce Type: replace-cross Abstract: Deploying domain-specialized large language models on resource-constrained hardware motivates reducing both adaptation cost and the number of retained weights. LoRA lowers adaptation cost but leaves a dense backbone, while separate pruning can discard weights that become important during adaptation. We propose EfficientXpert, a framework that co-adapts low-rank updates and sparse support. Its ForeSight Mask scores weights through a downstream reconstruction surrogate using the evolving LoRA-augmented weights; Partial Brain Surgeon (PBS) realigns the adapter through a closed-form correction. At 40% sparsity, the strongest variant retains 98.55--108.99% of dense-LoRA aggregate performance across the evaluated LLaMA settings. On Qwen3-8B, adaptation adds 17.9--27.0% training time and at most 4.8% peak GPU memory. Our analysis connects the domain- and rank-dependent effects of PBS to its constrained reconstruction objective, providing guidance for selecting adapter recovery capacity. Together, these results demonstrate efficient production of sparse, domain-specialized experts. Code is available at https://github.com/TagoreZhao/Efficient_Domain_Adaptation.
Abstract:Deploying domain-specialized large language models on resource-constrained hardware motivates reducing both adaptation cost and the number of retained weights. LoRA lowers adaptation cost but leaves a dense backbone, while separate pruning can discard weights that become important during adaptation. We propose EfficientXpert, a framework that co-adapts low-rank updates and sparse support. Its ForeSight Mask scores weights through a downstream reconstruction surrogate using the evolving LoRA-augmented weights; Partial Brain Surgeon (PBS) realigns the adapter through a closed-form correction. At 40% sparsity, the strongest variant retains 98.55--108.99% of dense-LoRA aggregate performance across the evaluated LLaMA settings. On Qwen3-8B, adaptation adds 17.9--27.0% training time and at most 4.8% peak GPU memory. Our analysis connects the domain- and rank-dependent effects of PBS to its constrained reconstruction objective, providing guidance for selecting adapter recovery capacity. Together, these results demonstrate efficient production of sparse, domain-specialized experts. Code is available at this https URL.
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2511.19935 [cs.LG]
(or arXiv:2511.19935v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2511.19935 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 243 cuvinte
Subiecte conexe
[AINews] not much happened today
Latent Space · 18 sept. 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
Distilled Rapid Embedding Transfer (DRET): Parameter-Efficient Biomedical Domain Adaptation via Priority-Based Embedding Transfer
arXiv · 4 sept. 2026
Building Blocks for Foundation Model Training and Inference on AWS
Hugging Face · 11 mai 2026