aiqbench analysis
OSPD addresses the critical challenge of maintaining persona consistency in role-playing language models by introducing an on-policy self-distillation framework that eliminates reliance on external teachers or reward models. The method leverages asymmetric information and role-aware divergence switching to align model confidence with character-critical tokens, demonstrating superior performance over supervised fine-tuning and multi-turn reinforcement learning baselines. This advancement represents a significant step toward more stable and coherent AI agents capable of sustained, faithful character embodiment without distribution mismatch issues.
OSPD: On-Policy Self-Distillation for Persona-Consistent Dialogue
arXiv:2609.34418v1 Announce Type: new Abstract: Maintaining persona consistency across multi-turn dialogues remains a core challenge for role-playing language models. Off-policy distillation from external teachers incurs distribution mismatch that compounds across dialogue turns, while reinforcement learning struggles with reward ambiguity inherent in subjective persona fidelity. We propose OSPD, an on-policy self-distillation framework where the same model serves as both teacher and student under asymmetric information: the teacher receives a complete character profile while the student sees only a brief summary, and the student generates trajectories from its own policy. We find that teacher confidence in role-playing dialogue exhibits a bimodal structure---sharply peaked at character-critical tokens yet diffuse at generic utterances---and introduce role-aware divergence switching to match this structure. A progressive trait masking curriculum further forces staged internalization of character knowledge along semantic dimensions. Experiments on CharacterBench, CharacterEval, and SocialBench show that OSPD substantially improves persona consistency over supervised fine-tuning and multi-turn RL baselines, without requiring any external teacher or reward model.
Abstract:Maintaining persona consistency across multi-turn dialogues remains a core challenge for role-playing language models. Off-policy distillation from external teachers incurs distribution mismatch that compounds across dialogue turns, while reinforcement learning struggles with reward ambiguity inherent in subjective persona fidelity. We propose OSPD, an on-policy self-distillation framework where the same model serves as both teacher and student under asymmetric information: the teacher receives a complete character profile while the student sees only a brief summary, and the student generates trajectories from its own policy. We find that teacher confidence in role-playing dialogue exhibits a bimodal structure---sharply peaked at character-critical tokens yet diffuse at generic utterances---and introduce role-aware divergence switching to match this structure. A progressive trait masking curriculum further forces staged internalization of character knowledge along semantic dimensions. Experiments on CharacterBench, CharacterEval, and SocialBench show that OSPD substantially improves persona consistency over supervised fine-tuning and multi-turn RL baselines, without requiring any external teacher or reward model.
Accepted to EMNLP 2026. 22 pages, including references and appendices
Artificial Intelligence (cs.AI)
arXiv:2609.34418 [cs.AI]
(or arXiv:2609.34418v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.34418 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 259 words
Related coverage
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · Oct 16, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · Sep 29, 2026
Some Interesting Papers on RLVR
LessWrong · Jun 9, 2026