analiza aiqbench
This research demonstrates that reinforcement learning post-training for code LLMs can be effectively conducted offline, significantly reducing the computational overhead associated with generating new code samples and managing GPU-CPU communication. By leveraging existing datasets, this approach enables substantial performance improvements in zero-shot code generation across various model sizes without requiring extensive online sampling infrastructure. The findings offer a practical pathway to optimize training efficiency while maintaining high functional correctness in code generation tasks.
Performance, Efficiency and Collapse -- Advantages and Challenges in Offline Post-training of Code LLMs
arXiv:2609.11956v1 Announce Type: new Abstract: Post-training with reinforcement learning (RL) is a critical phase in the development of code-generating large language models (LLMs), as it ensures adherence to instructions and the production of functionally correct code. This process typically requires computationally intensive code sample generation from Transformer-based LLMs and substantial GPU-CPU communication for sequence verification. To address these computational challenges, this work examines whether RL-based post-training can be performed entirely offline by leveraging existing datasets rather than generating new samples. The findings indicate that, with only a few hours of training, zero-shot code generation performance of LLMs can be substantially improved without online sampling. Additionally, offline RL produces performance gains across models ranging from 0.5B to 7B parameters, although the extent of improvement varies among model families.
Abstract:Post-training with reinforcement learning (RL) is a critical phase in the development of code-generating large language models (LLMs), as it ensures adherence to instructions and the production of functionally correct code. This process typically requires computationally intensive code sample generation from Transformer-based LLMs and substantial GPU-CPU communication for sequence verification. To address these computational challenges, this work examines whether RL-based post-training can be performed entirely offline by leveraging existing datasets rather than generating new samples. The findings indicate that, with only a few hours of training, zero-shot code generation performance of LLMs can be substantially improved without online sampling. Additionally, offline RL produces performance gains across models ranging from 0.5B to 7B parameters, although the extent of improvement varies among model families.
Machine Learning (cs.LG)
arXiv:2609.11956 [cs.LG]
(or arXiv:2609.11956v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.11956 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 214 cuvinte
Subiecte conexe
News
Quantum Computing Report · 7 iun. 2026
[AINews] Reality Checks on AI News (Yegge shuts down Gas Town, Databricks’ +60% Astra cost)
Latent Space · 17 sept. 2026
[AINews] not much happened today
Latent Space · 18 sept. 2026
Problems of Proliferation
LessWrong · 29 sept. 2026