analiza aiqbench
This paper finds that on-policy distillation, a common way to train smaller language models, can raise answer accuracy while making the model less responsive to the document it is supposed to answer from. The authors show this with matched document edits that change the supported answer, and they propose Evidence-Paired OPD, which recovers part of the lost evidence responsiveness on QASPER and PeerQA. The results are from a single model family and two datasets, so readers should treat the gains as a targeted training-objective result rather than a general fix.
Better Answers, Weaker Evidence Control: Evidence Bypass in On-Policy Distillation
arXiv:2611.01967v1 Announce Type: cross Abstract: On-policy distillation (OPD) can improve answer accuracy while weakening responsiveness to supporting evidence. We investigate this separation in scientific-document question answering through matched document interventions that change the supported answer while preserving the question and unrelated context. We identify evidence bypass: a student retains an originally correct answer after the document changes to support a different one. On QASPER with a Qwen3-4B student, standard OPD raises original-document accuracy by 3.4 percentage points, lowers counterfactual accuracy by 5.3 points, and increases bypass by 15.7 points; the same directional pattern appears on PeerQA. Controlled perturbations localize the shift to answer-critical evidence, and bypass grows with the student's pre-distillation closed-book preference for the original answer. A factorized study of teacher targets, privileged information, divergence, and supervision density finds that explicitly supplying the teacher with annotated answer-critical spans yields the largest joint improvement in evidence response and claim support. These findings motivate Evidence-Paired OPD (EP-OPD), which combines on-policy supervision on matched documents supporting different answers to the same question with an answer-level margin that favors each document's supported answer over its paired alternative. Across both datasets and two student sizes, EP-OPD recovers 54-66% of standard OPD's counterfactual-accuracy loss while preserving 80-85% of its original-document accuracy gain. Our results establish evidence responsiveness as a distinct training and evaluation target alongside answer accuracy. Our code is publicly available at: https://github.com/RuoranXu/EP-OPD.
Abstract:On-policy distillation (OPD) can improve answer accuracy while weakening responsiveness to supporting evidence. We investigate this separation in scientific-document question answering through matched document interventions that change the supported answer while preserving the question and unrelated context. We identify evidence bypass: a student retains an originally correct answer after the document changes to support a different one. On QASPER with a Qwen3-4B student, standard OPD raises original-document accuracy by 3.4 percentage points, lowers counterfactual accuracy by 5.3 points, and increases bypass by 15.7 points; the same directional pattern appears on PeerQA. Controlled perturbations localize the shift to answer-critical evidence, and bypass grows with the student's pre-distillation closed-book preference for the original answer. A factorized study of teacher targets, privileged information, divergence, and supervision density finds that explicitly supplying the teacher with annotated answer-critical spans yields the largest joint improvement in evidence response and claim support. These findings motivate Evidence-Paired OPD (EP-OPD), which combines on-policy supervision on matched documents supporting different answers to the same question with an answer-level margin that favors each document's supported answer over its paired alternative. Across both datasets and two student sizes, EP-OPD recovers 54-66% of standard OPD's counterfactual-accuracy loss while preserving 80-85% of its original-document accuracy gain. Our results establish evidence responsiveness as a distinct training and evaluation target alongside answer accuracy. Our code is publicly available at: this https URL.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2611.01967 [cs.CL]
(or arXiv:2611.01967v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.01967 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
figure_sources/README.txt
figure_sources/overview_standalone.tex
figure_sources/overview_tikz.tex
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 327 cuvinte
Subiecte conexe
Med-OPD: Improving Medical Vision-Language Models via Evidence-Aware On-Policy Distillation
arXiv · 21 iul. 2026
[Paper] Distillation for Incrimination and Distillation for Capabilities
LessWrong · 9 oct. 2026
DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Agent Memory
arXiv · 30 sept. 2026
MeSD: Multi-Evidence Self-Distillation for VideoLLM
arXiv · 7 oct. 2026