aiqbench analysis
This research addresses the critical training-deployment mismatch in LLMs by introducing CALM, a post-training framework that explicitly integrates diverse inference-time controllers into the learning loop. By treating reasoning strategies like Chain-of-Thought or debate as dynamic interaction protocols rather than fixed patterns, this approach significantly improves model generalization across varied workflows without requiring separate fine-tuning for each controller type. The work represents a substantive advance in machine learning methodology, focusing on how models can adapt to evolving inference architectures through multi-task reinforcement learning and modular reasoning decomposition.
Training Language Models to Cooperate with Inference-Time Controllers
arXiv:2607.23771v1 Announce Type: new Abstract: Large language model (LLM) performance increasingly depends not only on the base model, but also on the inference-time controller used to organize reasoning. Existing post-training methods, however, typically optimize for a single fixed interaction pattern, despite real deployments relying on diverse controllers such as Chain-of-Thought, self-consistency, debate, planning, and verification pipelines. This creates a training--deployment mismatch and limits transfer to new workflows. We introduce CALM (Controller-Aware Language Models), a post-training framework that explicitly places controllers in the training loop. We formulate controller-aware post-training as multi-task reinforcement learning over controller-induced interaction protocols, where controllers are compositions of reusable local reasoning modules. This structure also induces a module-level decomposition of mixed-controller training under a turn-level GRPO objective, enabling a systematic study of controller and module-aware training strategies. We evaluate CALM on held-out controller compositions and broader controller shifts, showing that controller-aware post-training improves generalization across inference-time workflows beyond single-controller optimization.
Abstract:Large language model (LLM) performance increasingly depends not only on the base model, but also on the inference-time controller used to organize reasoning. Existing post-training methods, however, typically optimize for a single fixed interaction pattern, despite real deployments relying on diverse controllers such as Chain-of-Thought, self-consistency, debate, planning, and verification pipelines. This creates a training--deployment mismatch and limits transfer to new workflows. We introduce CALM (Controller-Aware Language Models), a post-training framework that explicitly places controllers in the training loop. We formulate controller-aware post-training as multi-task reinforcement learning over controller-induced interaction protocols, where controllers are compositions of reusable local reasoning modules. This structure also induces a module-level decomposition of mixed-controller training under a turn-level GRPO objective, enabling a systematic study of controller and module-aware training strategies. We evaluate CALM on held-out controller compositions and broader controller shifts, showing that controller-aware post-training improves generalization across inference-time workflows beyond single-controller optimization.
Artificial Intelligence (cs.AI)
arXiv:2607.23771 [cs.AI]
(or arXiv:2607.23771v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2607.23771 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 242 words
Related coverage
2026 BAIR Graduate Showcase
BAIR Blog · Jul 1, 2026
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026
Lessons from scaling teams of coding agents: reach the frontier at 30% of the cost
Cohere · Oct 14, 2026
Do AI models assist with human rights violations?
LessWrong · Sep 29, 2026