aiqbench analysis
This research addresses the critical gap in dialectal machine translation by introducing SiLTT, a new benchmark specifically designed to capture linguistic variations like Polish-Silesian that standard datasets miss. The study demonstrates that rule-based systems currently outperform both general neural models and fine-tuned commercial baselines such as TranslateGemma on these specialized tasks, highlighting the need for domain-specific approaches over pure scale. By releasing their curated dataset and best-performing model, the authors provide essential resources to advance robustness in low-resource dialectal settings.
Precision over Scale: A Polish-Silesian Benchmark and a Translation System Outperforming Open-Source and Commercial Models
arXiv:2610.01082v1 Announce Type: new Abstract: Dialectal machine translation remains challenging due to limited data and strong linguistic variation not captured by standard benchmarks, which often assume standardized and well-edited text. We study Polish-Silesian MT using neural and rule-based systems, evaluating on SiLTT - a new Pol-Szl testset, alongside established BOUQuET and FLORES benchmarks. Results show our rule-based system is consistently strongest on SiLTT and BOUQuET datasets and that TranslateGemma fine-tuned on a curated dataset improves over strong neural baselines but does not surpass the rule-based system in dialectal settings. We release SiLTT and our best neural model to support further research.
Abstract:Dialectal machine translation remains challenging due to limited data and strong linguistic variation not captured by standard benchmarks, which often assume standardized and well-edited text. We study Polish-Silesian MT using neural and rule-based systems, evaluating on SiLTT - a new Pol-Szl testset, alongside established BOUQuET and FLORES benchmarks. Results show our rule-based system is consistently strongest on SiLTT and BOUQuET datasets and that TranslateGemma fine-tuned on a curated dataset improves over strong neural baselines but does not surpass the rule-based system in dialectal settings. We release SiLTT and our best neural model to support further research.
Computation and Language (cs.CL)
arXiv:2610.01082 [cs.CL]
(or arXiv:2610.01082v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.01082 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 191 words
