aiqbench analysis
This research addresses a critical misalignment in web agent architectures by introducing "predicted-state matching," a training objective that ensures world models generate representations discriminative enough for downstream reward ranking. By leveraging branching trajectories from WebArena, the study demonstrates that this approach significantly outperforms standard supervised next-state prediction in both action selection and end-to-end task success. The work advances the reliability of autonomous agents by optimizing the specific signal required for test-time computation rather than generic state prediction.
Discriminative World Models for Web Agents
arXiv:2609.02885v1 Announce Type: cross Abstract: Recent web agents use world models for test-time action selection by sampling candidate actions, predicting the resulting web states, and ranking them with a ranker model or a Process Reward Model (PRM). These world models are typically trained via supervised next-state prediction to generate fixed representations like HTML or AXTree snapshots. However, this objective is misaligned with the downstream ranker, which relies on predicted states being discriminative across candidates to accurately score them. To address this, we introduce predicted-state matching, a training objective where the predicted representation must distinguish the true resulting state from those reached by alternative actions. We train these models using a branching web-agent dataset derived from WebArena Go-Browse trajectories, where every decision point contains multiple alternative actions and their resulting states. Experiments on our held-out predicted-state matching benchmark show that our approach outperforms world models trained with supervised next-state prediction. We further show that our approach improves PRM-style action ranking on WebPRMBench compared with action-only PRMs and PRMs augmented with supervised-next-state world models. Finally, on WebArena-Lite, using our world model for test-time action selection improves end-to-end task success. Our project page is available at: https://dhruvpendharkar.github.io/dwm/.
Abstract:Recent web agents use world models for test-time action selection by sampling candidate actions, predicting the resulting web states, and ranking them with a ranker model or a Process Reward Model (PRM). These world models are typically trained via supervised next-state prediction to generate fixed representations like HTML or AXTree snapshots. However, this objective is misaligned with the downstream ranker, which relies on predicted states being discriminative across candidates to accurately score them. To address this, we introduce predicted-state matching, a training objective where the predicted representation must distinguish the true resulting state from those reached by alternative actions. We train these models using a branching web-agent dataset derived from WebArena Go-Browse trajectories, where every decision point contains multiple alternative actions and their resulting states. Experiments on our held-out predicted-state matching benchmark show that our approach outperforms world models trained with supervised next-state prediction. We further show that our approach improves PRM-style action ranking on WebPRMBench compared with action-only PRMs and PRMs augmented with supervised-next-state world models. Finally, on WebArena-Lite, using our world model for test-time action selection improves end-to-end task success. Our project page is available at: this https URL.
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2609.02885 [cs.AI]
(or arXiv:2609.02885v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.02885 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 288 words
Related coverage
[AINews] not much happened today
Latent Space · Sep 18, 2026
Do AI models assist with human rights violations?
LessWrong · Sep 29, 2026
Import AI 464: Fables writes GPU kernels; AI automation; and analog computation
Import AI · Jul 6, 2026
Self-Modeling Interventions Modulate Emergent Misalignment
LessWrong · Oct 5, 2026