analiza aiqbench
PartitionSel introduces a mathematically grounded approach to cross-domain minibatch selection that balances convergence speed with data coverage using partition-matroid constraints. By coupling per-domain budgets through a single validation-guided utility, the method reduces selection redundancy and gradient conflicts during fine-tuning of LLMs like Qwen2.5 and Llama-3. This technique offers provable approximation guarantees and robust empirical gains over existing baselines, addressing the computational inefficiencies of current proxy-model-dependent strategies.
Minibatch Selection via Partition Matroid Constrained Gradient Matching
arXiv:2606.07954v1 Announce Type: new Abstract: Training large language models (LLMs) on heterogeneous data requires selecting minibatches that balance convergence speed with coverage across domains. Existing methods either select samples independently within each domain or rely on computationally expensive proxy models to learn continuous domain weights. We propose PartitionSel, a cross-domain minibatch selection approach that maximizes a validation-guided gradient-matching utility under per-domain budgets encoded as a partition-matroid constraint. By coupling the per-domain budgets through a single utility, PartitionSel is designed to reduce redundancy in selections across domains. The proposed objective is weakly submodular and admits an orthogonal matching pursuit algorithm with provable approximation guarantees. Empirically, we evaluate PartitionSel for minibatch selection during the fine-tuning of Qwen2.5 and Llama-3 on MetaMathQA and Mol-Instructions. PartitionSel achieves robust gains over per-domain and domain-agnostic baselines on both benchmarks. It also reduces the number of conflicting gradient pairs within each batch, indicating that the cross-domain coupling translates into more compatible training updates.
Abstract:Training large language models (LLMs) on heterogeneous data requires selecting minibatches that balance convergence speed with coverage across domains. Existing methods either select samples independently within each domain or rely on computationally expensive proxy models to learn continuous domain weights. We propose PartitionSel, a cross-domain minibatch selection approach that maximizes a validation-guided gradient-matching utility under per-domain budgets encoded as a partition-matroid constraint. By coupling the per-domain budgets through a single utility, PartitionSel is designed to reduce redundancy in selections across domains. The proposed objective is weakly submodular and admits an orthogonal matching pursuit algorithm with provable approximation guarantees. Empirically, we evaluate PartitionSel for minibatch selection during the fine-tuning of Qwen2.5 and Llama-3 on MetaMathQA and Mol-Instructions. PartitionSel achieves robust gains over per-domain and domain-agnostic baselines on both benchmarks. It also reduces the number of conflicting gradient pairs within each batch, indicating that the cross-domain coupling translates into more compatible training updates.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
I.2.6; G.1.6; G.2.1
arXiv:2606.07954 [cs.LG]
(or arXiv:2606.07954v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.07954 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 251 cuvinte
Subiecte conexe
Dropless MoE Training in JAX with Primus-Turbo
AMD · 10 iun. 2026
Robust Active Learning for Few-Shot Example Selection in Text-to-SQL
arXiv · 30 sept. 2026
J++ Lens: Jacobian Filtering Enables More Faithful Workspace Lenses
LessWrong · 8 oct. 2026
Learning Which Correspondences to Trust: Confidence-Weighted Event-Camera Localization in LiDAR Maps
arXiv · 9 oct. 2026