analiza aiqbench
This research introduces EDIT, a two-phase training framework that leverages internal model signals to diagnose and correct reasoning errors in LLM graders, ensuring judgments remain grounded in rubrics rather than just predicting scores. By penalizing harmful belief drifts while allowing exploration, the method achieves consistent performance gains over standard supervised fine-tuning and reinforcement learning baselines across diverse grading benchmarks. The core contribution lies in shifting from opaque score prediction to transparent, evidence-based credit assignment, which is critical for deploying reliable automated evaluation systems.
EDIT: Evidence-Diagnosed Intervention Training for Rule-Faithful LLM Grading
arXiv:2606.06350v1 Announce Type: new Abstract: Reliable rubric grading requires more than accurate score prediction. Each judgement must be grounded in the mark scheme and evidence from the student answer. Existing credit-assignment and intervention methods, primarily designed for self-contained reasoning tasks such as mathematics reasoning, struggle in this setting because they do not identify where grading reasoning goes wrong or how the model's belief about the final mark changes during reasoning. We propose Evidence-Diagnosed Intervention Training (EDIT), a two-phase framework for training more rubric-faithful LLM graders. First, EDIT-SFT locates problematic reasoning steps using internal model signals: posterior belief over the final mark and input-grounding scores. It then revises only these local steps with help from a rubric checklist. Second, EDIT-RL calibrates the grader with belief-guided reward shaping, penalising large harmful belief drifts while still allowing helpful exploration. Experiments on two real-world, multi-subject grading benchmarks demonstrate that EDIT consistently outperforms strong supervised fine-tuning and reinforcement learning baselines on both in-domain and out-of-domain splits, with ablation studies confirming that internal-state diagnostics drive these gains.
Abstract:Reliable rubric grading requires more than accurate score prediction. Each judgement must be grounded in the mark scheme and evidence from the student answer. Existing credit-assignment and intervention methods, primarily designed for self-contained reasoning tasks such as mathematics reasoning, struggle in this setting because they do not identify where grading reasoning goes wrong or how the model's belief about the final mark changes during reasoning. We propose Evidence-Diagnosed Intervention Training (EDIT), a two-phase framework for training more rubric-faithful LLM graders. First, EDIT-SFT locates problematic reasoning steps using internal model signals: posterior belief over the final mark and input-grounding scores. It then revises only these local steps with help from a rubric checklist. Second, EDIT-RL calibrates the grader with belief-guided reward shaping, penalising large harmful belief drifts while still allowing helpful exploration. Experiments on two real-world, multi-subject grading benchmarks demonstrate that EDIT consistently outperforms strong supervised fine-tuning and reinforcement learning baselines on both in-domain and out-of-domain splits, with ablation studies confirming that internal-state diagnostics drive these gains.
Computation and Language (cs.CL)
arXiv:2606.06350 [cs.CL]
(or arXiv:2606.06350v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.06350 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 261 cuvinte
Subiecte conexe
EDIT: Evidence-Diagnosed Intervention Training for Rule-Faithful LLM Grading
arXiv · 4 sept. 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · 18 sept. 2026
Game Design Creativity evals
LessWrong · 12 oct. 2026
Improving CoT Monitorability of Evaluation Awareness via Verbalization Training
LessWrong · 2 oct. 2026