analiza aiqbench
This paper addresses critical integrity issues in multimodal physics reasoning models by exposing systematic distortions caused by data contamination, translation artifacts, and biased benchmarking practices. The authors release a comprehensive verifier system that includes a three-stage audit for train/test boundary certification, a binary reward mechanism to prevent exploitation of partial credits, and a dual-layer judging harness to ensure score reliability. By providing open-source tools that validate the training corpus and evaluation metrics, this work establishes a necessary baseline for trustworthy progress in scientific AI reasoning.
Physics-R1: An Audited Olympiad Corpus and Released Verifiers for Visual Physics Reasoning
arXiv:2605.14040v2 Announce Type: replace Abstract: Trackable improvement in multimodal physics reasoning rests on a training-and-evaluation system that is itself rarely verified: the corpora a model trains on, the reward it is optimized against, and the benchmarks and judges that score it. We audit this system end to end and find that standard construction practices systematically distort measurement: contamination slips past n-gram deduplication, translation degrades problems, saturated multiple-choice formats overstate capability, partial-credit training rewards are easier to exploit than to earn, and open-ended grading silently depends on the choice of judge. Left unverified, these distortions inflate reported progress and leak test knowledge into training. We answer with a released verifier system: a three-stage contamination audit that certifies the train/test boundary behind an audited multimodal training corpus and a held-out olympiad benchmark; a binary answer verifier that supplies the reinforcement-learning training reward; and an answer-judging harness that brackets every open-ended score between a deterministic strict layer and a large-language-model liberal layer. All per-record verdicts are released and cross-checked against an independent open-weight judge, whose substitution shifts absolute scores but preserves the sign of every base-to-trained lift. Training against the binary answer verifier confirms the certified corpus supports training: a reference recipe lifts an 8B open-source base by 18.3 points on the held-out benchmark across three seeds. The simple binary reward also beats a dense partial-credit variant on three of four open-ended benchmarks, tying the fourth. All verifiers, verdicts, and datasets are public.
Abstract:Trackable improvement in multimodal physics reasoning rests on a training-and-evaluation system that is itself rarely verified: the corpora a model trains on, the reward it is optimized against, and the benchmarks and judges that score it. We audit this system end to end and find that standard construction practices systematically distort measurement: contamination slips past n-gram deduplication, translation degrades problems, saturated multiple-choice formats overstate capability, partial-credit training rewards are easier to exploit than to earn, and open-ended grading silently depends on the choice of judge. Left unverified, these distortions inflate reported progress and leak test knowledge into training. We answer with a released verifier system: a three-stage contamination audit that certifies the train/test boundary behind an audited multimodal training corpus and a held-out olympiad benchmark; a binary answer verifier that supplies the reinforcement-learning training reward; and an answer-judging harness that brackets every open-ended score between a deterministic strict layer and a large-language-model liberal layer. All per-record verdicts are released and cross-checked against an independent open-weight judge, whose substitution shifts absolute scores but preserves the sign of every base-to-trained lift. Training against the binary answer verifier confirms the certified corpus supports training: a reference recipe lifts an 8B open-source base by 18.3 points on the held-out benchmark across three seeds. The simple binary reward also beats a dense partial-credit variant on three of four open-ended benchmarks, tying the fourth. All verifiers, verdicts, and datasets are public.
10 pages, 3 tables. Project page: this https URL
Computation and Language (cs.CL)
I.2.7; I.2.10; I.2.6
arXiv:2605.14040 [cs.CL]
(or arXiv:2605.14040v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.14040 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 340 cuvinte
