analiza aiqbench
SLIM introduces a method to reduce the computational cost of merging large language models by using structured evaluation points to predict optimal mixing coefficients. By constructing a quadratic surrogate function, this approach minimizes the number of expensive benchmark evaluations required to identify high-performing model combinations. The technique demonstrates improved prediction fidelity over random designs, offering a more efficient path for optimizing multi-expert mixtures within constrained budgets.
SLIM: Simplex-Lattice Interpolation Merging
arXiv:2610.01037v1 Announce Type: new Abstract: Optimizing merging coefficients for large language models can require many costly benchmark evaluations. We propose \textbf{Simplex-Lattice Interpolation Merging (SLIM)}, which constructs a quadratic surrogate of aggregate performance on the coefficient simplex using a classical mixture design. Evaluations of individual experts and equal-weight pairs determine the surrogate with the minimum number of measurements needed to identify a general quadratic on this domain. SLIM then optimizes the surrogate without further target-metric evaluations. Experiments on two model architectures demonstrate accurate prediction of unseen multi-expert mixtures and competitive merge performance under limited evaluation budgets. Matched-budget comparisons show that structured evaluation points improve prediction fidelity over random designs, including those using regularized fitting.
Abstract:Optimizing merging coefficients for large language models can require many costly benchmark evaluations. We propose \textbf{Simplex-Lattice Interpolation Merging (SLIM)}, which constructs a quadratic surrogate of aggregate performance on the coefficient simplex using a classical mixture design. Evaluations of individual experts and equal-weight pairs determine the surrogate with the minimum number of measurements needed to identify a general quadratic on this domain. SLIM then optimizes the surrogate without further target-metric evaluations. Experiments on two model architectures demonstrate accurate prediction of unseen multi-expert mixtures and competitive merge performance under limited evaluation budgets. Matched-budget comparisons show that structured evaluation points improve prediction fidelity over random designs, including those using regularized fitting.
Machine Learning (cs.LG)
arXiv:2610.01037 [cs.LG]
(or arXiv:2610.01037v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2610.01037 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 202 cuvinte
Subiecte conexe
Mixture-Trained Merging for Unified Multi-Objective Models
arXiv · 2 oct. 2026
IonQ and partners show practical applications and future roadmap at IEEE 2026
Quantum Zeitgeist · 14 sept. 2026
SLEM: Spectrum-Aware Latent Expert Merging
arXiv · 15 oct. 2026
Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective
arXiv · 21 iul. 2026