analiza aiqbench
SORT introduces a repair mechanism for reinforcement learning methods like GRPO, addressing their tendency to stall on hard prompts where standard rollouts fail. By deriving plans from reference solutions and weighting tokens based on plan conditioning, the method converts uniform imitation failures into selective, structure-aware learning signals. This approach demonstrates significant performance gains across multiple reasoning benchmarks, particularly benefiting weaker model architectures.
Selective Off-Policy Reference Tuning with Plan Guidance
arXiv:2605.11505v3 Announce Type: replace Abstract: Reinforcement learning with verifiable rewards helps reasoning, but GRPO-style methods stall on hard prompts where all sampled rollouts fail. SORT adds a repair update for those failures without changing rollout generation: it derives a plan from the reference solution, compares token probabilities with and without that plan, and gives higher weight to tokens that become more predictable under plan conditioning. This turns all-wrong prompts into selective, structure-aware learning signals instead of uniform imitation. Across three backbones and eight reasoning benchmarks, SORT improves over GRPO and guidance baselines, with largest gains on weaker models.
Abstract:Reinforcement learning with verifiable rewards helps reasoning, but GRPO-style methods stall on hard prompts where all sampled rollouts fail. SORT adds a repair update for those failures without changing rollout generation: it derives a plan from the reference solution, compares token probabilities with and without that plan, and gives higher weight to tokens that become more predictable under plan conditioning. This turns all-wrong prompts into selective, structure-aware learning signals instead of uniform imitation. Across three backbones and eight reasoning benchmarks, SORT improves over GRPO and guidance baselines, with largest gains on weaker models.
Artificial Intelligence (cs.AI)
arXiv:2605.11505 [cs.AI]
(or arXiv:2605.11505v3 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2605.11505 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 185 cuvinte
Subiecte conexe
A Red Line and Oversight Framework for Government AI Contracts
AI Alignment Forum · 18 iul. 2026
Introducing Gemma 4 models on Amazon Bedrock
AWS · 15 iun. 2026
Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker HyperPod
AWS · 4 sept. 2026
Evaluate skill-equipped agents with Strands Evals and Amazon Bedrock AgentCore
AWS · 22 sept. 2026