analiza aiqbench
This arXiv paper proposes Thinking-Free Policy Initialization (TFPI), a method for reinforcement learning with verifiable rewards that discards chain-of-thought content before RL training to cut compute and token use while reaching strong benchmark scores. The results are reported on a 4B model and benchmarks like AIME24 and LiveCodeBench, so readers should weigh them against the specific setup and compute budget rather than treat them as general claims. For ML practitioners, the practical takeaway is a cheaper training warm-start for reasoning models, pending independent replication.
Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
arXiv:2509.26226v3 Announce Type: replace-cross Abstract: Reinforcement Learning with Verifiable Reward (RLVR) effectively solves complex tasks but demands extremely long context lengths during training, leading to substantial computational costs. While multi-stage training can partially mitigate this, starting with overly short contexts often causes irreversible performance degradation, ultimately failing to reduce overall training compute significantly. In this paper, we introduce Thinking-Free Policy Initialization (TFPI), a simple yet effective adaptation to RLVR that bridges long Chain-of-Thought (CoT) distillation and standard RLVR. TFPI employs a simple ThinkingFree operation, explicitly discarding the thinking content via a direct append, to reduce token usage during inference. Training with ThinkingFree-adapted inputs improves performance and lowers token consumption, even in the original slow-thinking mode. Extensive experiments across various benchmarks have shown that TFPI accelerates RL convergence, achieves a higher performance ceiling, and yields more token-efficient reasoning models without specialized rewards or complex training designs. With TFPI only, we train a 4B model to reach 89.0% accuracy on AIME24 and 65.5% on LiveCodeBench using less than 4K H20 hours.
Abstract:Reinforcement Learning with Verifiable Reward (RLVR) effectively solves complex tasks but demands extremely long context lengths during training, leading to substantial computational costs. While multi-stage training can partially mitigate this, starting with overly short contexts often causes irreversible performance degradation, ultimately failing to reduce overall training compute significantly. In this paper, we introduce Thinking-Free Policy Initialization (TFPI), a simple yet effective adaptation to RLVR that bridges long Chain-of-Thought (CoT) distillation and standard RLVR. TFPI employs a simple ThinkingFree operation, explicitly discarding the thinking content via a direct </think> append, to reduce token usage during inference. Training with ThinkingFree-adapted inputs improves performance and lowers token consumption, even in the original slow-thinking mode. Extensive experiments across various benchmarks have shown that TFPI accelerates RL convergence, achieves a higher performance ceiling, and yields more token-efficient reasoning models without specialized rewards or complex training designs. With TFPI only, we train a 4B model to reach 89.0% accuracy on AIME24 and 65.5% on LiveCodeBench using less than 4K H20 hours.
ICLR 2026 camera-ready version. 25 pages. Corrected author name
Machine Learning (cs.LG); Computation and Language (cs.CL)
arXiv:2509.26226 [cs.LG]
(or arXiv:2509.26226v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2509.26226 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 270 cuvinte
Subiecte conexe
Runway’s WorldPrompt and the Engineering of Real-Time Worlds
Latent Space · 25 sept. 2026
Frontier post-training recipe review with Finbarr Timbers
Interconnects · 16 iun. 2026
Open Distillation of Hereditary Traits
AI Alignment Forum · 14 iul. 2026
Sen. Adam Schiff on AI regulation, free speech, and impeaching Trump one more time
The Verge · 5 oct. 2026