aiqbench analysis
This research addresses a critical flaw in AI evaluation where models reward rhetorical flair over scientific merit by introducing the "Rhetorical Robustness" benchmark. The proposed SciCore architecture mitigates this issue by averaging full-manuscript judgments with assessments based on an extracted, structured science core to ensure content-normalized review. This work establishes a new standard for trustworthy AI reviewers that prioritizes stability across rewrites while maintaining strong discrimination between papers of varying quality.
A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review
arXiv:2609.39027v1 Announce Type: new Abstract: AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimization over scientific improvement. We formulate Rhetorical Robustness as the joint requirement of stability across content-preserving rewrites and discrimination across papers. We introduce RobustReview, a controlled full-manuscript benchmark with 1,260 manuscript versions, and evaluate 30 reviewer configurations. The benchmark reveals false robustness, where low rewrite sensitivity coincides with score collapse across papers, and shows that human alignment and rhetorical robustness rank reviewers differently. Moreover, the evaluated content-focused prompting protocol does not consistently improve robustness across backbones. Motivated by these findings, we introduce SciCore, a dual-branch reviewer that averages a full-manuscript judgment with a judgment based on an extracted, structured science core. This design combines manuscript-level assessment with a content-normalized view intended to reduce rhetorical sensitivity. In our primary GPT-5.5 comparison, SciCore achieves a leading joint stability-discrimination profile among the benchmarked reviewers while maintaining competitive human alignment. These results identify rhetorical robustness as a distinct evaluation target and demonstrate the potential of science-core review to improve it.
Abstract:AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimization over scientific improvement. We formulate Rhetorical Robustness as the joint requirement of stability across content-preserving rewrites and discrimination across papers. We introduce RobustReview, a controlled full-manuscript benchmark with 1,260 manuscript versions, and evaluate 30 reviewer configurations. The benchmark reveals false robustness, where low rewrite sensitivity coincides with score collapse across papers, and shows that human alignment and rhetorical robustness rank reviewers differently. Moreover, the evaluated content-focused prompting protocol does not consistently improve robustness across backbones. Motivated by these findings, we introduce SciCore, a dual-branch reviewer that averages a full-manuscript judgment with a judgment based on an extracted, structured science core. This design combines manuscript-level assessment with a content-normalized view intended to reduce rhetorical sensitivity. In our primary GPT-5.5 comparison, SciCore achieves a leading joint stability-discrimination profile among the benchmarked reviewers while maintaining competitive human alignment. These results identify rhetorical robustness as a distinct evaluation target and demonstrate the potential of science-core review to improve it.
35 pages, 2 figures, 20 tables. Accepted (Oral) at AI-Native Academia @ NeurIPS 2026
Computation and Language (cs.CL)
arXiv:2609.39027 [cs.CL]
(or arXiv:2609.39027v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.39027 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 284 words
Related coverage
State of the Field: AI for Epistemics and Coordination
LessWrong · Oct 2, 2026
How independent researchers could investigate AI propensities after misalignment incidents
METR · Jul 28, 2026
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026
The standard for agents you can trust: Lessons from the federal front lines
Snorkel AI Blog · Jun 5, 2026