aiqbench analysis
This paper introduces Constraint-Margin DPO (CM-DPO), a novel fine-tuning method that addresses the binary nature of standard Direct Preference Optimization by incorporating continuous margins based on constraint violation severity. By separating hard and soft constraints through a lexicographic objective, the approach enables smaller 8B models to match multi-agent systems in latency while significantly outperforming GPT-4o on complex planning benchmarks. The technique effectively mitigates style biases and improves reasoning capabilities without relying on massive scaling factors.
CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning
arXiv:2610.09219v1 Announce Type: cross Abstract: Direct Preference Optimization (DPO) treats all constraint violations equally: a $1 budget overshoot and a $1,000 overshoot induce the same training signal. It is also susceptible to length and style bias when preference pairs come from different model families. We introduce Constraint-Margin DPO (CM-DPO), which replaces DPO's binary preference signal with a continuous margin derived from a deterministic symbolic verifier and scaled by violation severity. Hard and soft constraints are separated through a lexicographic objective, ensuring hard constraints are never traded off against preferences. To supply CM-DPO with bias-reduced training pairs, we generate preference data through procedurally generated constraint profiles (DCCG) and minimal-edit distillation from a reasoning teacher (RT-MED), within a framework we call SynPlan-R. On TravelPlanner, NaturalPlan, and out-of-distribution PlanBench, an 8B model fine-tuned with CM-DPO achieves 89.2% pass rate and 93.4% solve rate, matching multi-agent systems at 13x lower latency while outperforming GPT-4o on unseen Blocksworld by 9.2 points.
Authors:Rabimba Karanjai, Qun Gu, Hemanth Hegadehalli Madhavarao, Wenhuan Sun, Xiaojiao Yu, Suryabhan Singh Hada, Libin N. George, Uma Kona, Richard Williamson, Linsey Pang, Prakhar Mehrotra
Abstract:Direct Preference Optimization (DPO) treats all constraint violations equally: a $1 budget overshoot and a $1,000 overshoot induce the same training signal. It is also susceptible to length and style bias when preference pairs come from different model families. We introduce Constraint-Margin DPO (CM-DPO), which replaces DPO's binary preference signal with a continuous margin derived from a deterministic symbolic verifier and scaled by violation severity. Hard and soft constraints are separated through a lexicographic objective, ensuring hard constraints are never traded off against preferences. To supply CM-DPO with bias-reduced training pairs, we generate preference data through procedurally generated constraint profiles (DCCG) and minimal-edit distillation from a reasoning teacher (RT-MED), within a framework we call SynPlan-R. On TravelPlanner, NaturalPlan, and out-of-distribution PlanBench, an 8B model fine-tuned with CM-DPO achieves 89.2% pass rate and 93.4% solve rate, matching multi-agent systems at 13x lower latency while outperforming GPT-4o on unseen Blocksworld by 9.2 points.
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv:2610.09219 [cs.AI]
(or arXiv:2610.09219v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.09219 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 197 words
Related coverage
Coherent (COHR) Q4 2026 Earnings Call Transcript
fool.com · Aug 20, 2026
[AINews] not much happened today
Latent Space · Oct 3, 2026
Stock Market Today Highlights, Sept 2: Sensex ends 373 pts lower at 76, 570, Nifty below 24, 000; Asian Paints, HDFC Bank fall most
thehindubusinessline.com · Sep 2, 2026
Concrete Evaluations to Investigate the OpenAI Model That Hacked Hugging Face
AI Alignment Forum · Aug 3, 2026