aiqbench analysis
Reasoning models offer the most promise for exploitation tasks but remain too slow and expensive for practical deployment, while non-reasoning models with tool use can improve medium-difficulty performance despite still underperforming simple linear regression. The primary viable application identified is using LLMs to explore large action spaces with inherent semantics by suggesting suitable candidates. Ultimately, current LLMs fail to outperform basic statistical baselines in exploitation scenarios, even in non-linear settings.
Should You Use Your Large Language Model to Explore or Exploit?
arXiv:2502.00225v4 Announce Type: replace Abstract: We evaluate the ability of the current generation of large language models (LLMs) to help a decision-making agent facing an exploration-exploitation tradeoff. While previous work has largely study the ability of LLMs to solve combined exploration-exploitation tasks, we take a more systematic approach and use LLMs to explore and exploit in silos in various (contextual) bandit tasks. We find that reasoning models show the most promise for solving exploitation tasks, although they are still too expensive or too slow to be used in many practical settings. Motivated by this, we study tool use and in-context summarization using non-reasoning models. We find that these mitigations may be used to substantially improve performance on medium-difficulty tasks, however even then, all LLMs we study perform worse than a simple linear regression, even in non-linear settings. On the other hand, we find that LLMs do help at exploring large action spaces with inherent semantics, by suggesting suitable candidates to explore.
Abstract:We evaluate the ability of the current generation of large language models (LLMs) to help a decision-making agent facing an exploration-exploitation tradeoff. While previous work has largely study the ability of LLMs to solve combined exploration-exploitation tasks, we take a more systematic approach and use LLMs to explore and exploit in silos in various (contextual) bandit tasks. We find that reasoning models show the most promise for solving exploitation tasks, although they are still too expensive or too slow to be used in many practical settings. Motivated by this, we study tool use and in-context summarization using non-reasoning models. We find that these mitigations may be used to substantially improve performance on medium-difficulty tasks, however even then, all LLMs we study perform worse than a simple linear regression, even in non-linear settings. On the other hand, we find that LLMs do help at exploring large action spaces with inherent semantics, by suggesting suitable candidates to explore.
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2502.00225 [cs.LG]
(or arXiv:2502.00225v4 [cs.LG] for this version)
Full text under the publisher's open licence · 174 words
Related coverage
Frontier Risk Report (February to March 2026)
METR · May 19, 2026
Frontier AI Risk Report (February–March 2026)
METR · May 19, 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · May 19, 2026
LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
arXiv · Jul 31, 2026