analiza aiqbench
This study demonstrates that rubric-aware, multitask fine-tuning of transformer models significantly outperforms general-purpose LLMs in aligning automated grading with instructor behavior for introductory C++ assignments. By incorporating distribution-matching objectives and soft labels, the approach achieves lower mean absolute error and better fidelity to empirical grade distributions than accuracy-optimized baselines. The findings highlight the importance of calibration-aware training strategies over pure accuracy maximization for specific educational evaluation tasks.
Leveraging BART to Assess CS1 C++ Programming Assignments using Rubric-based Criteria
De la arXiv
Text integralarXiv:2606.03814v1 Announce Type: new Abstract: This paper investigates rubric-aware, multitask fine-tuning of transformer models for automated grading of introductory C++ programming assignments, with the goal of producing grade predictions that better reflect instructor grading behavior than general-purpose LLMs. Using multi-semester CS1 data, student submissions are paired with numeric scores, letter-grade buckets, and assignment rubrics, then preprocessed into unified sequences for transformer input. A BART encoder-decoder with LoRA adaptation is trained to jointly predict numeric grades and grade buckets, augmented with a distribution-matching term to align predicted and empirical grade distributions, an evaluation dimension often overlooked in prior work.
Experiments compare single-task and multitask training, hard one-hot versus fuzzy and boundary-based soft labels, and rubric versus no-rubric conditions, with additional T5 and pairwise-pretrained variants. Results show that multitask BART with boundary-based soft labels and rubric context achieves lower mean absolute error and stronger grade-distribution alignment than single-task, hard-label, or code-only baselines. Fully fine-tuned T5 further improves distributional fidelity, while pairwise pretraining reduces numeric error at the cost of minority-class sensitivity. Collectively, the findings suggest that calibration-aware, rubric-guided training produces more instructor-like grading behavior than accuracy-optimized alternatives.
Abstract:This paper investigates rubric-aware, multitask fine-tuning of transformer models for automated grading of introductory C++ programming assignments, with the goal of producing grade predictions that better reflect instructor grading behavior than general-purpose LLMs. Using multi-semester CS1 data, student submissions are paired with numeric scores, letter-grade buckets, and assignment rubrics, then preprocessed into unified sequences for transformer input. A BART encoder-decoder with LoRA adaptation is trained to jointly predict numeric grades and grade buckets, augmented with a distribution-matching term to align predicted and empirical grade distributions, an evaluation dimension often overlooked in prior work.
Experiments compare single-task and multitask training, hard one-hot versus fuzzy and boundary-based soft labels, and rubric versus no-rubric conditions, with additional T5 and pairwise-pretrained variants. Results show that multitask BART with boundary-based soft labels and rubric context achieves lower mean absolute error and stronger grade-distribution alignment than single-task, hard-label, or code-only baselines. Fully fine-tuned T5 further improves distributional fidelity, while pairwise pretraining reduces numeric error at the cost of minority-class sensitivity. Collectively, the findings suggest that calibration-aware, rubric-guided training produces more instructor-like grading behavior than accuracy-optimized alternatives.
Artificial Intelligence (cs.AI)
arXiv:2606.03814 [cs.AI]
(or arXiv:2606.03814v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2606.03814 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 274 cuvinte
Subiecte conexe
A Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models
arXiv · 4 nov. 2026
SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
arXiv · 17 nov. 2026
A Semitechnical Interlude on the Equivalence of Probabilistic and Deterministic Solomonoff Induction
LessWrong · 2 sept. 2026
AgentProofBench: Benchmarking Reusable Verifiers for Web Agents
arXiv · 27 oct. 2026