analiza aiqbench
This research investigates the reliability of multilingual reasoning benchmarks by exposing how automatic translation errors can cause identical outputs for logically opposite premises, particularly in Swahili. The study highlights that current acceptance metrics fail to certify counterfactual benchmarks because unfaithful translations obscure the true logical relations between source and target languages. For AI researchers building cross-lingual datasets, this underscores the critical need for rigorous validation of translation fidelity before deploying multilingual reasoning models.
When Opposite Premises Translate Identically: A Source-Conditioned Diagnostic for Multilingual Commonsense Items
arXiv:2610.24428v1 Announce Type: new Abstract: Multilingual reasoning comparisons depend on translated premises preserving the relations in their English sources. Automatic translation scores can rank source-target pairs, but an acceptance decision alone cannot certify a counterfactual benchmark field. We test a source-contrast diagnostic for ACCORD premises: reverse the English polarity, translate both arms independently, count identical target outputs, and score each arm against its own source. Across 71 matched question identities at two containing-item depths, five target languages, and three translation checkpoints, Swahili has 63, 113, and 67 exact collisions among 710 pairs per checkpoint. At a human-reference-calibrated threshold, round-trip NLI passes both arms of 3 of the 113 NLLB-3.3B collisions, while LaBSE passes 78. Each collision proves at least one unfaithful translation, but does not identify its arm. These controlled measurements do not establish screening accuracy on spontaneous translation errors or a language-by-hop reasoning effect; the study has no native equivalence labels or answer-model evaluation.
Abstract:Multilingual reasoning comparisons depend on translated premises preserving the relations in their English sources. Automatic translation scores can rank source-target pairs, but an acceptance decision alone cannot certify a counterfactual benchmark field. We test a source-contrast diagnostic for ACCORD premises: reverse the English polarity, translate both arms independently, count identical target outputs, and score each arm against its own source. Across 71 matched question identities at two containing-item depths, five target languages, and three translation checkpoints, Swahili has 63, 113, and 67 exact collisions among 710 pairs per checkpoint. At a human-reference-calibrated threshold, round-trip NLI passes both arms of 3 of the 113 NLLB-3.3B collisions, while LaBSE passes 78. Each collision proves at least one unfaithful translation, but does not identify its arm. These controlled measurements do not establish screening accuracy on spontaneous translation errors or a language-by-hop reasoning effect; the study has no native equivalence labels or answer-model evaluation.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2610.24428 [cs.CL]
(or arXiv:2610.24428v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.24428 Focus to learn more arXiv-issued DOI via DataCite
selected_source_ids.csv
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 247 cuvinte