aiqbench analysis
The paper applies GRPO, a reinforcement learning method, to fine-tune 1.5B to 14B LLMs with retrieval tools (Wikipedia revisions, news summaries) for forecasting real-world events. It reports that a 1.5B Qwen 2.5 model beat Claude Sonnet 3.5 on cross-entropy against market-implied probabilities, but that is a single-dataset, self-reported comparison, so generalization and the scaling claims need independent replication. The paper's discussion of which forecasting tasks count as verifiable and how aleatoric uncertainty limits reward signals is the more reusable contribution for RL-for-LLM practitioners.
Reinforcement Learning for LLM-based Event Forecasting
arXiv:2606.15917v2 Announce Type: replace Abstract: We use Group Relative Policy Optimization (GRPO), a recently devised sample and memory efficient reinforcement learning method, to finetune pretrained LLMs in the range of 1.5B to 14B parameters equipped with the ability to get current information through the use of a Wikipedia revisions tool, or news summaries, to forecast real events beyond the knowledge cutoff of the LLM, as well as problems made to simulate different aspects of the dynamics of that training. We use the results of these experiments to comment on the scaling capability of LLMs for forecasting, as well as classify how judgmental forecasting fits into the verifiable/unverifiable domain taxonomy, considering the impact of the inherent aleatoric uncertainty when forecasting future events (e.g. the roll of a die). As a result of the GRPO training, we manage to bring a 1.5B parameter transformer (Qwen 2.5 1.5B) to forecasting performance superior to Claude Sonnet 3.5 over the same dataset as measured by cross entropy from the market agreed probabilities. We also discuss various dead ends on the path to this result.
Abstract:We use Group Relative Policy Optimization (GRPO), a recently devised sample and memory efficient reinforcement learning method, to finetune pretrained LLMs in the range of 1.5B to 14B parameters equipped with the ability to get current information through the use of a Wikipedia revisions tool, or news summaries, to forecast real events beyond the knowledge cutoff of the LLM, as well as problems made to simulate different aspects of the dynamics of that training. We use the results of these experiments to comment on the scaling capability of LLMs for forecasting, as well as classify how judgmental forecasting fits into the verifiable/unverifiable domain taxonomy, considering the impact of the inherent aleatoric uncertainty when forecasting future events (e.g. the roll of a die). As a result of the GRPO training, we manage to bring a 1.5B parameter transformer (Qwen 2.5 1.5B) to forecasting performance superior to Claude Sonnet 3.5 over the same dataset as measured by cross entropy from the market agreed probabilities. We also discuss various dead ends on the path to this result.
Machine Learning (cs.LG)
arXiv:2606.15917 [cs.LG]
(or arXiv:2606.15917v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.15917 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 266 words
Related coverage
Towards Sustainable Hydrogen Systems: Supply Chain Optimization with Model Predictive Control and Reinforcement Learning
arXiv · Sep 14, 2026
Do Your Own Research: Learning to Forecast by Learning to Search
arXiv · Oct 2, 2026
Learning Agentic Time Series Forecasting via Trajectory Distillation and Memory-Based Reinforcement Learning
arXiv · Oct 19, 2026
DAD4TS: Data-Augmentation-Oriented Diffusion Model for Time-Series Forecasting with Small-Scale Data
arXiv · Jun 3, 2026