analiza aiqbench
Policy Split introduces a dual-mode architecture for LLM reinforcement learning that separates task correctness from exploration without compromising model accuracy. By sharing parameters while applying distinct entropy regularization objectives, the method enables collaborative learning where high-entropy modes generate unique behavioral patterns to enhance creative task performance. This approach offers a promising direction for improving RL efficiency in general and creative domains compared to existing entropy-guided baselines.
Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization
arXiv:2604.11510v2 Announce Type: replace-cross Abstract: To encourage diverse exploration in reinforcement learning (RL) for large language models (LLMs) without compromising accuracy, we propose Policy Split, a novel paradigm that bifurcates the policy into normal and high-entropy modes with a high-entropy prompt. While sharing model parameters, the two modes undergo collaborative dual-mode entropy regularization tailored to distinct objectives. Specifically, the normal mode optimizes for task correctness, while the high-entropy mode incorporates a preference for exploration, and the two modes learn collaboratively. Extensive experiments demonstrate that our approach consistently outperforms established entropy-guided RL baselines across various model sizes in general and creative tasks. Further analysis reveals that Policy Split facilitates dual-mode exploration, where the high-entropy mode generates distinct behavioral patterns to the normal mode, providing unique learning signals.
Abstract:To encourage diverse exploration in reinforcement learning (RL) for large language models (LLMs) without compromising accuracy, we propose Policy Split, a novel paradigm that bifurcates the policy into normal and high-entropy modes with a high-entropy prompt. While sharing model parameters, the two modes undergo collaborative dual-mode entropy regularization tailored to distinct objectives. Specifically, the normal mode optimizes for task correctness, while the high-entropy mode incorporates a preference for exploration, and the two modes learn collaboratively. Extensive experiments demonstrate that our approach consistently outperforms established entropy-guided RL baselines across various model sizes in general and creative tasks. Further analysis reveals that Policy Split facilitates dual-mode exploration, where the high-entropy mode generates distinct behavioral patterns to the normal mode, providing unique learning signals.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2604.11510 [cs.CL]
(or arXiv:2604.11510v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2604.11510 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 220 cuvinte
Subiecte conexe
Fog of Love: Engineering Virtuous Agent Behavior with Affinity-based Reinforcement Learning in a Game Environment
arXiv · 5 iun. 2026
Google DeepMind report finds Frontier AI poses new cybersecurity and bio risks.
Quantum Zeitgeist · 4 aug. 2026
Language Unalignability: Why Some Concepts Resist Cross-Cultural Benchmark Evaluation
arXiv · 7 oct. 2026
Diversity Over Frequency: Rethinking Tool Use in Visual Chain-of-Thought Agents
arXiv · 3 iun. 2026