aiqbench analysis
This research introduces DiGA, a framework that decouples geometric components within task vectors to prevent cross-component interference during model merging. By orthogonally decomposing updates into distinct subspaces, the method preserves individual capability identities and mitigates performance degradation in consolidated models. The approach demonstrates improved efficacy across diverse architectures and tasks while remaining compatible with existing merging techniques.
Orthogonal Yet Coupled: Decoupling Geometric Components for Model Merging
arXiv:2609.37564v1 Announce Type: new Abstract: Merging pretrained models has emerged as an effective approach for consolidating diverse capabilities into a single unified model. However, prevailing merging methods typically treat each task vector as an indivisible merging unit, overlooking the heterogeneous geometric changes encoded within it. This treatment can induce cross-component coupling: when merging decisions are derived from statistics of the complete task vector, the geometric characteristics of one component may influence how another is selected, weighted, or combined, potentially degrading the quality of the merged model. To address this issue, we propose DiGA, a Disentangled Geometry-Aware model merging framework. Using the pretrained weights as a shared geometric reference, DiGA orthogonally decomposes each task vector into components corresponding to distinct geometric attributes. Rather than merging the task vectors as a whole, DiGA aggregates corresponding components independently within their respective subspaces and subsequently recombines them into a unified update. This component-wise formulation preserves the geometric identity of each component and prevents the characteristics of one component from interfering with the aggregation of another. Furthermore, DiGA can be incorporated into a broad range of existing model merging methods. Extensive experiments across diverse models, tasks, and merging methods demonstrate that DiGA improves merged-model performance and reduces capability degradation. Our repository is on https://github.com/wzj1718/DiGA.
Abstract:Merging pretrained models has emerged as an effective approach for consolidating diverse capabilities into a single unified model. However, prevailing merging methods typically treat each task vector as an indivisible merging unit, overlooking the heterogeneous geometric changes encoded within it. This treatment can induce cross-component coupling: when merging decisions are derived from statistics of the complete task vector, the geometric characteristics of one component may influence how another is selected, weighted, or combined, potentially degrading the quality of the merged model. To address this issue, we propose DiGA, a Disentangled Geometry-Aware model merging framework. Using the pretrained weights as a shared geometric reference, DiGA orthogonally decomposes each task vector into components corresponding to distinct geometric attributes. Rather than merging the task vectors as a whole, DiGA aggregates corresponding components independently within their respective subspaces and subsequently recombines them into a unified update. This component-wise formulation preserves the geometric identity of each component and prevents the characteristics of one component from interfering with the aggregation of another. Furthermore, DiGA can be incorporated into a broad range of existing model merging methods. Extensive experiments across diverse models, tasks, and merging methods demonstrate that DiGA improves merged-model performance and reduces capability degradation. Our repository is on this https URL.
Computation and Language (cs.CL)
arXiv:2609.37564 [cs.CL]
(or arXiv:2609.37564v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.37564 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 301 words
Related coverage
ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs
arXiv · Oct 2, 2026
DCRL: Decoupling and Coupling Reinforcement Learning via Policy-Reward Manifold Alignment
arXiv · Sep 24, 2026
Dual-Domain Manifold Modeling for Hyperspectral Image Fusion
arXiv · Jul 29, 2026
Boundary mutual information in double holography
arXiv · Aug 6, 2026