analiza aiqbench
This research introduces a framework for synthesizing high-quality supervision data for process reward models by precisely controlling error injection and verifying trajectory consistency. The approach demonstrates improved performance on logical and mathematical reasoning benchmarks while highlighting that fine-grained first-error localization remains a significant challenge. For practitioners, this underscores the critical need for verifiable, step-level data to effectively train models capable of robust reasoning.
Controllable and Verifiable Process Data Synthesis for Process Reward Models
arXiv:2605.02395v2 Announce Type: replace Abstract: Process reward models (PRMs) rely on high-quality process supervision data, yet existing construction methods often provide limited control over error location, error type, and trajectory consistency. We propose a controllable and verifiable framework for synthesizing process supervision data for PRMs. Our framework first constructs a correct symbolic reasoning chain, injects a template-aware error into an intermediate step, recomputes subsequent steps under the corrupted state, and verifies that the injected step is not derivable from its prefix. The resulting paired trajectories are prefix-invalid at the first error while remaining trajectory-consistent after symbolic recomputation, and are translated into aligned natural-language processes for PRM training and evaluation. Experiments show that the synthesized data improve Best-of-8 reranking on logical reasoning benchmarks and transfer to mathematical reasoning. Step-level evaluation further shows that first-error localization remains substantially more challenging than overall step classification, highlighting the need for fine-grained and verifiable process supervision.
Abstract:Process reward models (PRMs) rely on high-quality process supervision data, yet existing construction methods often provide limited control over error location, error type, and trajectory consistency. We propose a controllable and verifiable framework for synthesizing process supervision data for PRMs. Our framework first constructs a correct symbolic reasoning chain, injects a template-aware error into an intermediate step, recomputes subsequent steps under the corrupted state, and verifies that the injected step is not derivable from its prefix. The resulting paired trajectories are prefix-invalid at the first error while remaining trajectory-consistent after symbolic recomputation, and are translated into aligned natural-language processes for PRM training and evaluation. Experiments show that the synthesized data improve Best-of-8 reranking on logical reasoning benchmarks and transfer to mathematical reasoning. Step-level evaluation further shows that first-error localization remains substantially more challenging than overall step classification, highlighting the need for fine-grained and verifiable process supervision.
Artificial Intelligence (cs.AI)
arXiv:2605.02395 [cs.AI]
(or arXiv:2605.02395v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.02395 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 238 cuvinte
Subiecte conexe
When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion
arXiv · 24 iul. 2026
14 Human-in-the-Loop Patterns for AI Agents
Akka Blog · 4 nov. 2026
$Z^2$-ACT: End-to-End Verifiable Agentic Intent Control for Open 6G RAN
arXiv · 24 aug. 2026
SATQuest: A Verifier for Logical Reasoning Evaluation and Reinforcement Fine-Tuning of LLMs
arXiv · 21 iul. 2026