Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences

arXiv:2605.30100v2 Announce Type: replace Abstract: World models require state tracking, which is the ability to maintain a correct latent state across action sequences. Existing benchmarks are often synthetic or language-based, limiting their value as tests of structured state updates in realistic domains. We introduce Chess-World-Model, a large-scale state-tracking benchmark built from 10 million real chess games, where models predict the exact board state reached after a sequence of legal moves. Alongside a held-out real-game split, we include an out-of-distribution test set of uniformly random legal play both at its original trajectory lengths and after matching the human-game length distribution. This tests whether models learn the transition rules rather than shortcuts from common human positions. Prior theoretical and empirical work has shown that Transformers struggle to state-track, while input-dependent linear RNNs require expressive state-transition matrices to do so. We therefore benchmark a causal Transformer, block-diagonal SLiCE, Mamba-3, and Gated DeltaNet with negative eigenvalues under a matched interface and training protocol. The recurrent models strongly outperform the Transformer at approximately 3 and 8 million parameters. Real-game performance largely saturates by the 18 million parameter scale, but length-matched random evaluation continues to distinguish architectures through the largest tested scale of approximately 43 million parameters. Additionally, ablations show that less expressive state-transition mechanisms generally reduce performance on the out-of-distribution splits. Together, these results establish Chess-World-Model as a practical large-scale benchmark for state tracking that exposes failures model scale would otherwise conceal.