MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning

Abstract:Medical reasoning can reach a plausible answer while overlooking an unsafe or unsupported intermediate step. Process-Level Reward Models (PRMs) can verify such steps, yet existing PRM benchmarks largely target mathematics rather than patient-specific clinical errors. We introduce \textbf{MedPRMBench}, a step-level medical reasoning benchmark with 14 clinical error types, including five new and two clinically re-scoped types, and rule-defined severity grades. Built from seven medical QA sources via structured error construction and review by 86 physicians from Grade-A tertiary hospitals, it provides 6{,}500 evaluation questions, 13{,}000 reasoning chains, and 113{,}910 step labels, plus a leakage-clean 6{,}870-question training split. The best of 20 prompted critics reaches only 75.4\% PRMScore, and critics differ sharply in error-type profiles and operating points. A Qwen3-8B PRM trained on the split reaches 86.6\% PRMScore and 89.0\% strict first-error localization; in matched ablations, physician review raises error recall by 6.55 points and cuts clean-chain false alarms from 5.6\% to 2.1\%. The training signal replicates across model families; without further training, the PRM transfers to ProcessBench and PRMBench, exceeding six of nine math-trained PRMs on each.

Computation and Language (cs.CL)

arXiv:2604.17282 [cs.CL]

(or arXiv:2604.17282v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2604.17282 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.