Model What Matters: Decision-Guided Selective World Modeling for Language Agents

Abstract:Reinforcement learning provides a direct mechanism for optimizing task objectives in large language model-driven agents, but task rewards are typically sparse trajectory-level signals that only evaluate task success, without directly supervising action-conditioned environment transitions, leaving the model with limited explicit understanding of action consequences and environment dynamics. Recent work introduces auxiliary supervision through policy and world modeling co-training, yet typically treats complete environmental observations uniformly as world modeling targets, overlooking the varying decision relevance of observation content. We propose DGS-WM (Decision-Guided Selective World Modeling), a decision-guided policy-world modeling co-training approach for language agents. DGS-WM employs a Decision-Guided Observation Selection component that filters observation units with decision value as world modeling targets based on their support for subsequent policy responses, and a Confidence-Adaptive Weighting component that adaptively adjusts supervision strength according to the model's prediction confidence on selected targets. Together, these components form a decision-aware auxiliary world modeling loss, jointly optimized with the RL loss on the same policy model. Across multiple benchmarks covering single-hop/multi-hop question answering and interactive web tasks, DGS-WM achieves consistent improvements under different model scales and RL algorithms, demonstrating more stable and effective performance compared to full-observation world modeling methods.

Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)

arXiv:2610.19642 [cs.AI]

(or arXiv:2610.19642v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2610.19642 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

dgs_wm_core/FILES.sha256

dgs_wm_core/README.md

dgs_wm_core/openqa_agent/__init__.py

dgs_wm_core/openqa_agent/actions.py

dgs_wm_core/openqa_agent/data.py

dgs_wm_core/openqa_agent/environment.py

dgs_wm_core/openqa_agent/evaluation.py

dgs_wm_core/openqa_agent/prompts.py

dgs_wm_core/openqa_agent/retrieval/__init__.py

dgs_wm_core/openqa_agent/retrieval/client.py

dgs_wm_core/openqa_agent/retrieval/server.py

dgs_wm_core/openqa_agent/types.py

dgs_wm_core/openqa_agent/verl_adapter/__init__.py

dgs_wm_core/openqa_agent/verl_adapter/agent_loop.py

dgs_wm_core/openqa_agent/verl_adapter/data.py

dgs_wm_core/openqa_agent/verl_adapter/main_ppo_wm.py

dgs_wm_core/openqa_agent/verl_adapter/reward.py

dgs_wm_core/openqa_agent/verl_adapter/sentence_selector.py

dgs_wm_core/openqa_agent/verl_adapter/wm_targets.py

dgs_wm_core/openqa_agent/verl_adapter/wm_trainer.py

dgs_wm_core/patches/verl-v0.8.0-compact-lora-checkpoint.patch

dgs_wm_core/patches/verl-v0.8.0-vllm-lora-prompt-logprobs.patch

dgs_wm_core/patches/webshop-environment.patch

dgs_wm_core/pyproject.toml

dgs_wm_core/third_party/VERL_LICENSE.txt

dgs_wm_core/third_party/WebShop_LICENSE.md

dgs_wm_core/webshop_agent/__init__.py

dgs_wm_core/webshop_agent/actions.py

dgs_wm_core/webshop_agent/context.py

dgs_wm_core/webshop_agent/environment.py

dgs_wm_core/webshop_agent/episode.py

dgs_wm_core/webshop_agent/pages.py

dgs_wm_core/webshop_agent/prompts.py

dgs_wm_core/webshop_agent/protocol.py

dgs_wm_core/webshop_agent/refinement.py

dgs_wm_core/webshop_agent/remote.py

dgs_wm_core/webshop_agent/rollout.py

dgs_wm_core/webshop_agent/service.py

dgs_wm_core/webshop_agent/verl_adapter/__init__.py

dgs_wm_core/webshop_agent/verl_adapter/agent_loop.py

dgs_wm_core/webshop_agent/verl_adapter/data.py

dgs_wm_core/webshop_agent/verl_adapter/main_ppo.py

dgs_wm_core/webshop_agent/verl_adapter/response_selector.py

dgs_wm_core/webshop_agent/verl_adapter/trainer.py

(39 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.