Language Models that Think, Chat Better

arXiv:2509.20357v2 Announce Type: replace Abstract: Reinforcement learning with verifiable rewards (RLVR) trains language models to use long chain-of-thought reasoning (CoT) in domains like mathematics and code with rule-based verifiers. However, long CoT learned through RLVR does not generalize well to open-ended tasks -- such as writing essay outlines or making meal plans -- where humans reason routinely. This paper establishes the benefits of long CoT for general-purpose chat capabilities and introduces RL with Model-rewarded Thinking (RLMT)1, which pushes RLVR beyond verifiable domains. Using diverse real-world prompts, RLMT requires LMs to generate long CoT reasoning before responding, and optimizes them with online RL against a preference-based reward model used in RLHF. Across 40 training runs on Llama-3.1-8B and Qwen-2.5-7B (both base and instruct) and multiple optimization algorithms (DPO, PPO, and GRPO), RLMT consistently outperforms standard RLHF pipelines. This includes substantial gains of 3-7 points on three chat benchmarks (AlpacaEval2, WildBench, and ArenaHardV2), along with 1-3 point improvements on other tasks like creative writing and general knowledge. RLMT can also be applied directly to base models without an SFT stage, akin to DeepSeek-R1-Zero. Remarkably, with only 7K prompts, Llama-3.1-8B base trained with our RLMT recipe outperforms Llama-3.1-8B-Instruct post-trained with a complex multi-staged pipeline with 25M+ examples. We close with qualitative and quantitative analyses of how trained models plan their responses. Our results rethink the post-training pipeline and call upon future work to understand and employ thinking more broadly.

Abstract:Reinforcement learning with verifiable rewards (RLVR) trains language models to use long chain-of-thought reasoning (CoT) in domains like mathematics and code with rule-based verifiers. However, long CoT learned through RLVR does not generalize well to open-ended tasks -- such as writing essay outlines or making meal plans -- where humans reason routinely. This paper establishes the benefits of long CoT for general-purpose chat capabilities and introduces RL with Model-rewarded Thinking (RLMT)1, which pushes RLVR beyond verifiable domains. Using diverse real-world prompts, RLMT requires LMs to generate long CoT reasoning before responding, and optimizes them with online RL against a preference-based reward model used in RLHF. Across 40 training runs on Llama-3.1-8B and Qwen-2.5-7B (both base and instruct) and multiple optimization algorithms (DPO, PPO, and GRPO), RLMT consistently outperforms standard RLHF pipelines. This includes substantial gains of 3-7 points on three chat benchmarks (AlpacaEval2, WildBench, and ArenaHardV2), along with 1-3 point improvements on other tasks like creative writing and general knowledge. RLMT can also be applied directly to base models without an SFT stage, akin to DeepSeek-R1-Zero. Remarkably, with only 7K prompts, Llama-3.1-8B base trained with our RLMT recipe outperforms Llama-3.1-8B-Instruct post-trained with a complex multi-staged pipeline with 25M+ examples. We close with qualitative and quantitative analyses of how trained models plan their responses. Our results rethink the post-training pipeline and call upon future work to understand and employ thinking more broadly.

Computation and Language (cs.CL)

arXiv:2509.20357 [cs.CL]

(or arXiv:2509.20357v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2509.20357 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Language Models that Think, Chat Better

Subiecte conexe