analiza aiqbench
This paper proposes TADrop, a method for merging fine-tuned model checkpoints that sets sparsity levels per tensor, based on each tensor's weight distribution, instead of applying one global ratio. The reported gains are modest (up to about 5 points on some benchmarks), and they come from controlled comparisons against specific merging baselines. Readers building multi-task models or LoRA merges should check whether the data-free default holds up on their own architectures before adopting it.
One Size Does Not Fit All: Distribution-Guided Tensor-Wise Adaptive Sparsification for Model Merging
arXiv:2508.06163v2 Announce Type: replace-cross Abstract: Merging multiple task-specific fine-tuned checkpoints into a single multi-task model provides an attractive alternative to joint multi-task training. To mitigate cross-task interference, a common strategy is to sparsify task vectors before aggregation. However, many widely used methods rely on a uniform global sparsity ratio, overlooking differences in pruning sensitivity across tasks and parameter tensors. Consequently, such uniform sparsification may remove useful parameter updates from sensitive tensors while retaining redundant updates in others. To address this limitation, we propose TADrop, a plug-and-play sparsifier that independently determines tensor-specific keep ratios from their magnitude distributions using a scale-invariant quantile ratio. TADrop operates in a data-free manner by default, while offering optional lightweight module-level calibration when a small validation set is available. Experiments across vision, language, and multimodal benchmarks, including LLM and LoRA merging, demonstrate consistent improvements across multiple host merging methods, including a 4.99-point gain over EMR-Merging on RoBERTa GLUE. In many-task ViT-B/16 merging, the improvement increases with the number of tasks, reaching 2.26 points when merging 30 tasks. Further analyses show that TADrop preserves more task-associated neurons under matched overall sparsity. Together, these findings highlight the value of distribution-guided sparsification at the tensor level for improving model merging.
Abstract:Merging multiple task-specific fine-tuned checkpoints into a single multi-task model provides an attractive alternative to joint multi-task training. To mitigate cross-task interference, a common strategy is to sparsify task vectors before aggregation. However, many widely used methods rely on a uniform global sparsity ratio, overlooking differences in pruning sensitivity across tasks and parameter tensors. Consequently, such uniform sparsification may remove useful parameter updates from sensitive tensors while retaining redundant updates in others. To address this limitation, we propose TADrop, a plug-and-play sparsifier that independently determines tensor-specific keep ratios from their magnitude distributions using a scale-invariant quantile ratio. TADrop operates in a data-free manner by default, while offering optional lightweight module-level calibration when a small validation set is available. Experiments across vision, language, and multimodal benchmarks, including LLM and LoRA merging, demonstrate consistent improvements across multiple host merging methods, including a 4.99-point gain over EMR-Merging on RoBERTa GLUE. In many-task ViT-B/16 merging, the improvement increases with the number of tasks, reaching 2.26 points when merging 30 tasks. Further analyses show that TADrop preserves more task-associated neurons under matched overall sparsity. Together, these findings highlight the value of distribution-guided sparsification at the tensor level for improving model merging.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2508.06163 [cs.CL]
(or arXiv:2508.06163v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2508.06163 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 295 cuvinte
Subiecte conexe
Quantization and Pruning Methods to Make Your LLM Leaner
KDnuggets · 28 aug. 2026
ORBIT-2 based Weather and Climate Downscaling and Downscaled Global Forecasts on AMD Instinct
AMD · 8 iun. 2026
PIRAMID: Progress and Plans
LessWrong · 27 iul. 2026
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · 4 sept. 2026