FutureEval Spring Results: Pros Beat Bots, but the Gap is Nearly Gone

Main Takeaways Top Findings: Pro forecasters beat bots but without significance: Our team of 10 Metaculus Pro Forecasters outperformed the top-10 bot team by an average of 1.25 head-to-head points per question. But this edge is small relative to the question-to-question variation and not statistically significant (one-sided p = 0.247), so this sample doesn't give us enough evidence to confirm Pros were genuinely ahead this season. The bot team improved notably in the last year: The bot team’s head-to-head score against Pros was -11.3 in Q3 2024 (95% CI: [-21.8, -0.7]),  -8.9 in Q4 2024 (95% CI: [-18.8, 1]), -17.7 in Q1 2025 (95% CI: [-28.3, -7.0]), and then -20.03 in Q2 2025 (95% CI: [-28.63, -11.41]). This time the bot team’s score improved to -1.25 (95% CI: [-4.87, 2.37]), closing much of the gap. (Reminder: 0 would mean bots and Pros did equally well, and a positive number would mean bots beat Pros.) Nine of ten individual Pros beat every individual bot: We created a leaderboard where both Pros and bots competed together as individuals. In this comparison, 9 of the top 10 places would be taken by Pros. Two bots slightly edged out one of the Pros. Note that though this type of comparison is interesting, unlike the Pro vs Bot team comparison, the more individuals we compare, the higher the chance of one bot or Pro getting incorrectly ranked due to luck.  Other Takeaways: Bot team aggregation peaked around 10 bots and got worse afterwards. A team of the top two bots from the leaderboard performed better than the top bot alone. Groupings up to a size of 10 performed similarly well. Performance began to decline around a bot team of size 10. Merely increasing the size of a bot team does not automatically improve the ensemble’s forecasting. The highest-scoring models were OpenAI’s GPT-5.1 and above: Metaculus ran many bots that used the same prompt but different LLMs. The one using OpenAI’s GPT-5.1-high finished first among these, and 18th overall in the bot tournament. Based on a survey of our bot makers, among non-Metaculus bots, every one of the top 10 survey responders reported using GPT-5.2 or higher GPT-5.x models as part of their final prediction ensemble, with 9 of them using GPT-5.4. Though it was not significant, we found usage of GPT-5.4 to be the strongest explanatory marker for a bot’s performance in the tournament. Agent scaffolding continues to improve: For most of FutureEval history, a single-prompt non-agentic baseline bot has placed in the top 10 bots. In Q2 2025, our top baseline bot (using OpenAI o3) placed 2nd out of 96. In Fall 2025, the top baseline bot took 14th out of 134 (and a modified bot with agentic search took 10th). This season, the best baseline bot (using GPT 5.1) placed 18th out of 173. Since Fall 2025, there has been a clear shift in how well LLMs are able to leverage forecasting harnesses. No one leads the pack: None of the top 10 bots scored well enough to clearly differentiate from each other. Their average spot peer scores ranged from 13.2 to 18.9, and the average confidence interval of the top 10 bots was ~8.5 points wide. Students and hobbyists continued to perform well: 7 of the top 10 bot makers were hobbyists, including two of the top 3 bot makers. No search provider is clearly better than another: As in previous analyses, we have continued to find that switching out individual search providers doesn’t seem to cause a notable difference in forecasting performance. In the spring tournament, eleven Metac Bots used DeepSeek-R1 to forecast while using different search setups. Scores were close, and any differences could be attributed to noise.  Introduction In the first third of 2026 (January 7th through April 15th), Metaculus ran the fifth tournament in our FutureEval series (rebranded from AI Benchmarking).  It had two aims: 1) comparing AI bots to the best human forecasters and 2) evaluating the best forecasting bots against one another. This is the primary analysis post for Spring FutureEval, and it focuses especially on comparing the rankings of Pros and Bots. It is complementary to the Spring Bot Survey Analysis and the Spring Advice from Bot makers to Bot Makers Analysis. You can find our past research on the FutureEval resources page. There were 173 bots (111 excluding the ones run by Metaculus) forecasting on 297 scored questions about events resolving sometime in January through May 2026. (Typical examples: How many vehicles will Tesla deliver globally in Q1 2026? and Will OpenAI file an initial registration statement (S-1) with the SEC to launch an IPO, before March 15, 2026?) The non-Metaculus botmakers competed for a prize pool of $50,000. A team of 10 Metaculus Pro forecasters served as a human benchmark on 99 of the 297 questions against the bots that finished in the top 10 of the leaderboard in the bot-only questions. The forecasting questions spanned the entire range of categories found at the main Metaculus platform: Economy & Business, Politics, Ge