Zero-shot reasoning for simulating scholarly peer-review

Abstract:Scholarly publishing requires scalable scrutiny supported by auditable evidence. This paper presents a two-component benchmark of xPeer, the peer-review simulation engine delivered through the this http URL web front-end. The operational component analyzes 352 of 500 simulation records retained under stable-task criteria across disciplines and review modes. The human-reference component releases 1,108 version-1 F1000Research manuscript records with linked human reports and applies a prespecified two-human/two-xPeer comparison. Human-review text and recommendations remained outside the generation input, and source joining occurred after xPeer outputs had been persisted. This procedure defines workflow-level review withholding; prior model exposure falls outside the recorded design. Among 802 records with exactly two human reports, 271 contained two usable xPeer reviewer fields, giving a complete-pair availability rate of 33.8%. Under deterministic extraction rules, median manuscript-level report length was 1,889 words for xPeer and 763 for humans, while median concern count was 41 and 13, respectively. xPeer reports showed higher targeting, category coverage, and executability. Human reports showed higher explicit-reasoning language, lexical manuscript attestation, taxonomy-based scientific relevance, and lower mean within-source redundancy. Cross-source lexical concern matching and recommendation agreement were low. The evidence therefore defines distinct observable review profiles and a transparent reproducibility baseline. Scientific correctness of individual concerns, autonomous editorial use, and cross-system superiority require expert adjudication and common-protocol testing. The study-level dataset and exact version-pinned reproducibility record are archived on Zenodo

Artificial Intelligence (cs.AI); Emerging Technologies (cs.ET)

I.2.4; H.5.3; J.7; K.4.3

arXiv:2510.02027 [cs.AI]

(or arXiv:2510.02027v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2510.02027 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.