BOW: Training Language Models to Reason Over Plausible Next Words

arXiv:2506.13502v3 Announce Type: replace Abstract: Next-word prediction (NWP) trains language models against a single observed continuation, even though many contexts admit multiple plausible next words. Recent RL-based next-word reasoning methods make this tension explicit: they reward a model for producing a rationale that supports one context-conditioned continuation, which can turn a pre-existing preference into a confident, self-justifying trajectory. We introduce BOW, an RL framework that instead trains models to produce self-contained, neutral, and comprehensive descriptions of the plausible next-word space. BOW's core reward is mediated by the generated trajectory. The policy conditions on the full context, but a frozen scorer assigns the core reward from the trajectory alone, without receiving the original context as a separate input. The trajectory may restate relevant context; the bottleneck is the missing direct context-to-scorer path in the core reward. BOW-Reg adds a lightweight breadth regularizer around this core term to discourage premature collapse. Across ten general reasoning benchmarks, BOW remains competitive with the original instruction models and often outperforms trained baselines. On both backbones, BOW-Reg achieves the highest SharedRef correctness and the lowest HoWN-Simple single-sense collapse. Human evaluation further shows that BOW-Reg elicits broader next-word reasoning trajectories, while intrinsic NWP results show that these trajectories remain predictive.

Abstract:Next-word prediction (NWP) trains language models against a single observed continuation, even though many contexts admit multiple plausible next words. Recent RL-based next-word reasoning methods make this tension explicit: they reward a model for producing a rationale that supports one context-conditioned continuation, which can turn a pre-existing preference into a confident, self-justifying trajectory. We introduce BOW, an RL framework that instead trains models to produce self-contained, neutral, and comprehensive descriptions of the plausible next-word space. BOW's core reward is mediated by the generated trajectory. The policy conditions on the full context, but a frozen scorer assigns the core reward from the trajectory alone, without receiving the original context as a separate input. The trajectory may restate relevant context; the bottleneck is the missing direct context-to-scorer path in the core reward. BOW-Reg adds a lightweight breadth regularizer around this core term to discourage premature collapse. Across ten general reasoning benchmarks, BOW remains competitive with the original instruction models and often outperforms trained baselines. On both backbones, BOW-Reg achieves the highest SharedRef correctness and the lowest HoWN-Simple single-sense collapse. Human evaluation further shows that BOW-Reg elicits broader next-word reasoning trajectories, while intrinsic NWP results show that these trajectories remain predictive.

Computation and Language (cs.CL)

arXiv:2506.13502 [cs.CL]

(or arXiv:2506.13502v3 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2506.13502 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

aiqbench analysis

BOW: Training Language Models to Reason Over Plausible Next Words

Related coverage