analiza aiqbench
This research addresses the exploration challenges inherent in Large Language Model action spaces by introducing Instruction-Conditioned Exploration (ICE) paired with an Asymmetric Reinforcement Learning and Self-Distillation objective. The method successfully enhances mathematical reasoning performance on Qwen3-1.7B, demonstrating that leveraging pre-trained knowledge to generate diverse training experiences yields measurable gains over standard RL approaches like DAPO.
Instruction-Conditioned Exploration with Asymmetric Reinforcement Learning and Self-Distillation
arXiv:2608.02087v1 Announce Type: new Abstract: Post-training Large Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges distinct from classical RL, with implications for inducing exploration. New methods are required that leverage the broad knowledge and flexibility of pre-trained LLMs to deliberately generate diverse experience at training time. We propose Instruction-Conditioned Exploration (ICE), which supplements task prompts during training with one of several distinct instructions, increasing the coverage of behaviours attempted. To facilitate ICE, we propose Asymmetric-RL/SD, a combined Reinforcement Learning and Self-Distillation training objective, to transfer explored behaviours to the unconditioned test-time policy. ICE with the Asymmetric-RL/SD objective improves Qwen3-1.7B held-out pass@1 performance at $4$K response length on mathematical reasoning tasks by $5.0\%$ relative to training with DAPO, with improvement persisting at a longer 8K context.
Abstract:Post-training Large Language Models (LLMs) with Reinforcement Learning (RL) has become an important tool for improving model capabilities, but the LLM action-space structure introduces challenges distinct from classical RL, with implications for inducing exploration. New methods are required that leverage the broad knowledge and flexibility of pre-trained LLMs to deliberately generate diverse experience at training time. We propose Instruction-Conditioned Exploration (ICE), which supplements task prompts during training with one of several distinct instructions, increasing the coverage of behaviours attempted. To facilitate ICE, we propose Asymmetric-RL/SD, a combined Reinforcement Learning and Self-Distillation training objective, to transfer explored behaviours to the unconditioned test-time policy. ICE with the Asymmetric-RL/SD objective improves Qwen3-1.7B held-out pass@1 performance at $4$K response length on mathematical reasoning tasks by $5.0\%$ relative to training with DAPO, with improvement persisting at a longer 8K context.
Submitted to ACL Rolling Review (ARR) May 2026 cycle. OpenReview submission record at this https URL
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv:2608.02087 [cs.AI]
(or arXiv:2608.02087v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2608.02087 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 252 cuvinte
Subiecte conexe
Deconfusing Corrigibility
AI Alignment Forum · 17 oct. 2026
Game Design Creativity evals
LessWrong · 12 oct. 2026
Reward Laundering: LLMs Can Gain Unintended Behaviors by Deciding When to Earn Their Rewards
LessWrong · 31 iul. 2026
LLMs can improve CoT Controllability via Reward Laundering
LessWrong · 17 oct. 2026