aiqbench analysis
This research proposes a novel reinforcement learning approach to optimize tokenization boundaries directly within LLM architectures, moving away from hardcoded compression steps toward fully end-to-end training. By leveraging score function estimates and techniques like time discounting, the method achieves tighter theoretical guarantees and superior performance compared to previous straight-through estimation heuristics at scale. This advancement addresses a critical bottleneck in model efficiency, offering a more principled path for integrating discrete token decisions into continuous deep learning pipelines.
You Can Learn Tokenization End-to-End with Reinforcement Learning
arXiv:2602.13940v3 Announce Type: replace-cross Abstract: Tokenization is a hardcoded compression step which remains in the training pipeline of Large Language Models (LLMs), despite a general trend towards architectures becoming increasingly end-to-end. Prior work has shown promising results at scale in bringing this compression step inside the LLMs' architecture with heuristics to draw token boundaries, and also attempts to learn these token boundaries with straight-through estimates, which treat the problem of drawing discrete token boundaries as a continuous one. We show that these token boundaries can instead be learned using score function estimates, which have tighter theoretical guarantees due to directly optimizing the problem of drawing discrete token boundaries to minimize loss. We observe that techniques from reinforcement learning, such as time discounting, are necessary to reduce the variance of this score function sufficiently to make it practicable. We demonstrate that the resultant method outperforms prior proposed straight-through estimates, both qualitatively and quantitatively at the $100$ million parameter scale.
Abstract:Tokenization is a hardcoded compression step which remains in the training pipeline of Large Language Models (LLMs), despite a general trend towards architectures becoming increasingly end-to-end. Prior work has shown promising results at scale in bringing this compression step inside the LLMs' architecture with heuristics to draw token boundaries, and also attempts to learn these token boundaries with straight-through estimates, which treat the problem of drawing discrete token boundaries as a continuous one. We show that these token boundaries can instead be learned using score function estimates, which have tighter theoretical guarantees due to directly optimizing the problem of drawing discrete token boundaries to minimize loss. We observe that techniques from reinforcement learning, such as time discounting, are necessary to reduce the variance of this score function sufficiently to make it practicable. We demonstrate that the resultant method outperforms prior proposed straight-through estimates, both qualitatively and quantitatively at the $100$ million parameter scale.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
arXiv:2602.13940 [cs.LG]
(or arXiv:2602.13940v3 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2602.13940 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 248 words
Related coverage
Reflection’s Beam model hits 501 billion parameters
Quantum Zeitgeist · Oct 5, 2026
Granite 4.1 LLMs: How They’re Built
Hugging Face · Apr 29, 2026
Enabling Language-specific Reasoning in Multilingual Models with Reinforcement Learning
AMD · Jul 24, 2026
Agentic Reinforcement Learning with Self-Distilled Reward Shaping
arXiv · Aug 6, 2026