analiza aiqbench
PolyGloss addresses a critical gap in language documentation by jointly predicting morphological segmentation and glossing, resolving the interpretability issues that previously hindered the adoption of models like GlossLM. This multilingual seq2seq approach outperforms existing benchmarks on both tasks and demonstrates strong adaptability to new datasets via low-rank adaptation. The work offers a more trustworthy tool for linguists by ensuring predicted morpheme boundaries align accurately with generated glosses.
Massively Multilingual Joint Segmentation and Glossing
arXiv:2601.10925v3 Announce Type: replace Abstract: Automated interlinear gloss prediction with neural networks is a promising approach to accelerate language documentation efforts. However, while state-of-the-art models like GlossLM achieve high scores on glossing benchmarks, user studies with linguists have found critical barriers to the usefulness of such models in real-world scenarios. In particular, existing models typically generate morpheme-level glosses but assign them to whole words without predicting the actual morpheme boundaries, making the predictions less interpretable and thus untrustworthy to human annotators. We conduct the first study on neural models that jointly predict interlinear glosses and the corresponding morphological segmentation from raw text. We run experiments to determine the optimal way to train models that balance segmentation and glossing accuracy, as well as the alignment between the two tasks. We extend the training corpus of GlossLM and pretrain PolyGloss, a family of seq2seq multilingual models for joint segmentation and glossing that outperforms GlossLM on glossing and beats various open-source LLMs on segmentation, glossing, and alignment. In addition, we demonstrate that PolyGloss can be quickly adapted to a new dataset via low-rank adaptation.
Abstract:Automated interlinear gloss prediction with neural networks is a promising approach to accelerate language documentation efforts. However, while state-of-the-art models like GlossLM achieve high scores on glossing benchmarks, user studies with linguists have found critical barriers to the usefulness of such models in real-world scenarios. In particular, existing models typically generate morpheme-level glosses but assign them to whole words without predicting the actual morpheme boundaries, making the predictions less interpretable and thus untrustworthy to human annotators. We conduct the first study on neural models that jointly predict interlinear glosses and the corresponding morphological segmentation from raw text. We run experiments to determine the optimal way to train models that balance segmentation and glossing accuracy, as well as the alignment between the two tasks. We extend the training corpus of GlossLM and pretrain PolyGloss, a family of seq2seq multilingual models for joint segmentation and glossing that outperforms GlossLM on glossing and beats various open-source LLMs on segmentation, glossing, and alignment. In addition, we demonstrate that PolyGloss can be quickly adapted to a new dataset via low-rank adaptation.
Computation and Language (cs.CL)
arXiv:2601.10925 [cs.CL]
(or arXiv:2601.10925v3 [cs.CL] for this version)
Text integral sub licența deschisă a publicației · 188 cuvinte
Subiecte conexe
Putting sign language AI into users’ hands
Google DeepMind · 12 aug. 2026
Mapping Political-Elite Networks in Europe with a Multilingual Joint Entity-Relation Extraction Pipeline
arXiv · 27 iul. 2026
A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrieval
arXiv · 4 sept. 2026
Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programming
arXiv · 10 iun. 2026