aiqbench analysis
The paper finds that appending a failed move and its error message to an LLM agent's transcript can make small models (135M to 1.7B parameters) more likely to repeat that move, at least for illegal moves in Sokoban, Connect Four, and Othello. The effect varies by model and task, and the mechanism is only exploratory, but the finding is a concrete warning that common retry harnesses can backfire. Practitioners should test feedback formats empirically rather than assume that showing an error helps, and note that the effect largely disappears after DPO post-training.
Feedback That Backfires in Games: Do LLM Players Repeat the Move They Just Watched Fail?
arXiv:2610.30205v1 Announce Type: new Abstract: When an LLM agent's move fails, the harness writes the move and the failure message into the transcript and asks again. In tool calling, this record makes small models more likely to repeat the failed action. We test whether the same holds in games (Sokoban, Connect Four, Othello), where a move can fail by being illegal or by being legal but losing, both verified exactly. On fixed probe items built by perturbing a correct move, the record backfires: pooled over 8 instruction-tuned models (135M-1.7B) it makes the failed move more likely (corrective gain G = -1.38 nats per token), through the move's surface form, for illegal moves but not for losing ones, and not for every model. An exploratory analysis shows what sets the sign: after the record, the failed move's log-probability barely depends on how likely it was before (slope 0.18). Moves the model would not have played become more likely, and greedy decoding starts to produce them; moves it was about to play become less likely, but greedy decoding still repeats 38% of them. A description of the failure that omits the move string keeps the model's preference instead. Along the OLMo 2 1B post-training ladder the inversion is present in the base model and largely removed at the DPO stage. RAGEN's default models invert in RAGEN's own Sokoban format before any RL. A "do not repeat" instruction does nothing, a clean retry is the worst harness, and a decoder ban stops exact repeats but not illegal play. All measurements ran in float32 on a free Colab GPU, which first reproduced the original paper's number.
Title:Feedback That Backfires in Games: Do LLM Players Repeat the Move They Just Watched Fail?
Abstract:When an LLM agent's move fails, the harness writes the move and the failure message into the transcript and asks again. In tool calling, this record makes small models more likely to repeat the failed action. We test whether the same holds in games (Sokoban, Connect Four, Othello), where a move can fail by being illegal or by being legal but losing, both verified exactly. On fixed probe items built by perturbing a correct move, the record backfires: pooled over 8 instruction-tuned models (135M-1.7B) it makes the failed move more likely (corrective gain G = -1.38 nats per token), through the move's surface form, for illegal moves but not for losing ones, and not for every model. An exploratory analysis shows what sets the sign: after the record, the failed move's log-probability barely depends on how likely it was before (slope 0.18). Moves the model would not have played become more likely, and greedy decoding starts to produce them; moves it was about to play become less likely, but greedy decoding still repeats 38% of them. A description of the failure that omits the move string keeps the model's preference instead. Along the OLMo 2 1B post-training ladder the inversion is present in the base model and largely removed at the DPO stage. RAGEN's default models invert in RAGEN's own Sokoban format before any RL. A "do not repeat" instruction does nothing, a clean retry is the worst harness, and a decoder ban stops exact repeats but not illegal play. All measurements ran in float32 on a free Colab GPU, which first reproduced the original paper's number.
20 pages, 6 figures. Code, probe items and raw records: this https URL
Artificial Intelligence (cs.AI)
arXiv:2610.30205 [cs.AI]
(or arXiv:2610.30205v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.30205 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 389 words
Related coverage
Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail
arXiv · Aug 27, 2026
Managers are using AI to write performance reviews – and it shows
ZDNet · Oct 7, 2026
Character training can mitigate reward hacking, but can also make it harder to detect
LessWrong · Sep 28, 2026
The current bottleneck is political will, not research
AI Alignment Forum · Jul 11, 2026