aiqbench analysis
This study isolates the specific drivers behind domain balancing in multi-domain meeting summarization, determining whether gains stem from token distribution or total data volume when fine-tuning LLMs like Mistral-7B. The findings indicate that redistributing quality via balanced mixtures significantly boosts performance on minority domains without heavily penalizing majority ones, while also demonstrating the efficiency of pruning low-value transcript lines to reduce training costs. These insights provide practitioners with a concrete framework for optimizing imbalanced multi-domain datasets and selecting appropriate balancing units beyond simple example counts.
Token Distribution versus Data Volume: Domain Balancing in Multi-Domain Meeting Summarisation
arXiv:2608.15935v1 Announce Type: new Abstract: Jointly fine-tuning an LLM on meeting-summarisation corpora of widely varying size raises a question that prior work leaves confounded: when a domain-balanced training mixture helps, is the gain due to the distribution of tokens across domains, or merely to the volume of data seen? We disentangle these factors by constructing balanced and natural (native-proportional) token mixtures at matched token budgets (2-32M) over five English meeting corpora, fine-tuning Mistral-7B with QLoRA, and evaluating per domain. Balancing redistributes quality, improving the data-scarce minority domains at a low cost to the data-rich ones. The trade favours balancing whenever the minority domains matter: their share under proportional allocation is fixed at 1-2% regardless of budget, so matching balanced quality on those domains requires far more total data. We further find that pruning low-value transcript lines removes ~15% of tokens from the conversational corpora at no measurable cost, and that balancing by tokens is not the same as balancing by examples. A two-annotator study of 741 judge-labelled facts validates our fact-level evaluation. Together these results give practitioners a basis for deciding when to balance an imbalanced multi-domain mixture, and on what unit.
Abstract:Jointly fine-tuning an LLM on meeting-summarisation corpora of widely varying size raises a question that prior work leaves confounded: when a domain-balanced training mixture helps, is the gain due to the distribution of tokens across domains, or merely to the volume of data seen? We disentangle these factors by constructing balanced and natural (native-proportional) token mixtures at matched token budgets (2-32M) over five English meeting corpora, fine-tuning Mistral-7B with QLoRA, and evaluating per domain. Balancing redistributes quality, improving the data-scarce minority domains at a low cost to the data-rich ones. The trade favours balancing whenever the minority domains matter: their share under proportional allocation is fixed at 1-2% regardless of budget, so matching balanced quality on those domains requires far more total data. We further find that pruning low-value transcript lines removes ~15% of tokens from the conversational corpora at no measurable cost, and that balancing by tokens is not the same as balancing by examples. A two-annotator study of 741 judge-labelled facts validates our fact-level evaluation. Together these results give practitioners a basis for deciding when to balance an imbalanced multi-domain mixture, and on what unit.
Computation and Language (cs.CL)
arXiv:2608.15935 [cs.CL]
(or arXiv:2608.15935v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.15935 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 282 words
Related coverage
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · Sep 4, 2026
Presentation: Beyond Prompting: Context Engineering and Memory Management for AI Systems at Scale
InfoQ · Jun 10, 2026
AMD Instinct MI455X vs MI355X: A Technical Look at the Advancing AI 2026 Inference Numbers
AMD · Oct 8, 2026
The art and science of hyperparameter optimization on Amazon Nova Forge
AWS · Jun 2, 2026