Density-Matrix Reinforcement Learning

arXiv:2610.17214v1 Announce Type: new Abstract: What information does a reinforcement learning policy omit? At each state, a standard policy stores one probability per action. It has no place to record how actions relate to each other, such as which pairs go together and which conflict. We ask when this missing structure matters, and we give exact answers. We store the policy at each state as a density matrix, a positive semidefinite, unit-trace matrix borrowed from quantum information theory. We use it here as a purely classical data structure, computed on ordinary classical hardware. The diagonal holds the usual action probabilities. The off-diagonal entries, called coherences, are new parameters. This gives density-matrix reinforcement learning (DMRL). Coherences change outcomes in two settings. First, suppose a policy is regularized toward a coherent reference policy using matrix relative entropy. The optimum then has coherences that no diagonal reference policy produces, and its action probabilities can depend on the reference policy's coherences. This is relevant to fine-tuning of large language models (LLMs) with a Kullback--Leibler penalty. Second, suppose the agent chooses a unit-vector action and the reward is a quadratic function of that vector. This could arise when an LLM judge scores a weighted consolidation of reasoning paths, or when coupled robotic sensors each respond to a mix of actions. Then some coherent policies earn strictly more expected reward than every classical policy with the same action probabilities, and we give the margin in closed form. In a two-action example, both policies assign probability $0.5$ to each action, yet the classical one earns expected reward $0$ while the coherent one earns $1$, the maximum. Outside these two settings, DMRL matches or transfers the classical results. This is a theory paper. All results are proved, and we report no empirical evaluation.

Abstract:What information does a reinforcement learning policy omit? At each state, a standard policy stores one probability per action. It has no place to record how actions relate to each other, such as which pairs go together and which conflict. We ask when this missing structure matters, and we give exact answers. We store the policy at each state as a density matrix, a positive semidefinite, unit-trace matrix borrowed from quantum information theory. We use it here as a purely classical data structure, computed on ordinary classical hardware. The diagonal holds the usual action probabilities. The off-diagonal entries, called coherences, are new parameters. This gives density-matrix reinforcement learning (DMRL). Coherences change outcomes in two settings. First, suppose a policy is regularized toward a coherent reference policy using matrix relative entropy. The optimum then has coherences that no diagonal reference policy produces, and its action probabilities can depend on the reference policy's coherences. This is relevant to fine-tuning of large language models (LLMs) with a Kullback--Leibler penalty. Second, suppose the agent chooses a unit-vector action and the reward is a quadratic function of that vector. This could arise when an LLM judge scores a weighted consolidation of reasoning paths, or when coupled robotic sensors each respond to a mix of actions. Then some coherent policies earn strictly more expected reward than every classical policy with the same action probabilities, and we give the margin in closed form. In a two-action example, both policies assign probability $0.5$ to each action, yet the classical one earns expected reward $0$ while the coherent one earns $1$, the maximum. Outside these two settings, DMRL matches or transfers the classical results. This is a theory paper. All results are proved, and we report no empirical evaluation.

Machine Learning (cs.LG)

arXiv:2610.17214 [cs.LG]

(or arXiv:2610.17214v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.17214 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Density-Matrix Reinforcement Learning

Subiecte conexe