Skip to content
aiqbenchAI & Quantum Intelligence
NewsInsightsBenchmarksSkillsAsk AILearnAbout
...Loading your session
NewsInsightsBenchmarksSkillsAsk AILearnAbout
Back to list
aiqbench takeAI Products

aiqbench analysis

This study demonstrates that rubric-aware, multitask fine-tuning of transformer models significantly outperforms general-purpose LLMs in aligning automated grading with instructor behavior for introductory C++ assignments. By incorporating distribution-matching objectives and soft labels, the approach achieves lower mean absolute error and better fidelity to empirical grade distributions than accuracy-optimized baselines. The findings highlight the importance of calibration-aware training strategies over pure accuracy maximization for specific educational evaluation tasks.
1
Source coveringarXiv

Leveraging BART to Assess CS1 C++ Programming Assignments using Rubric-based Criteria

arXiv·June 3, 2026·3 min read

From arXiv

Full text

arXiv:2606.03814v1 Announce Type: new Abstract: This paper investigates rubric-aware, multitask fine-tuning of transformer models for automated grading of introductory C++ programming assignments, with the goal of producing grade predictions that better reflect instructor grading behavior than general-purpose LLMs. Using multi-semester CS1 data, student submissions are paired with numeric scores, letter-grade buckets, and assignment rubrics, then preprocessed into unified sequences for transformer input. A BART encoder-decoder with LoRA adaptation is trained to jointly predict numeric grades and grade buckets, augmented with a distribution-matching term to align predicted and empirical grade distributions, an evaluation dimension often overlooked in prior work.

Experiments compare single-task and multitask training, hard one-hot versus fuzzy and boundary-based soft labels, and rubric versus no-rubric conditions, with additional T5 and pairwise-pretrained variants. Results show that multitask BART with boundary-based soft labels and rubric context achieves lower mean absolute error and stronger grade-distribution alignment than single-task, hard-label, or code-only baselines. Fully fine-tuned T5 further improves distributional fidelity, while pairwise pretraining reduces numeric error at the cost of minority-class sensitivity. Collectively, the findings suggest that calibration-aware, rubric-guided training produces more instructor-like grading behavior than accuracy-optimized alternatives.

Abstract:This paper investigates rubric-aware, multitask fine-tuning of transformer models for automated grading of introductory C++ programming assignments, with the goal of producing grade predictions that better reflect instructor grading behavior than general-purpose LLMs. Using multi-semester CS1 data, student submissions are paired with numeric scores, letter-grade buckets, and assignment rubrics, then preprocessed into unified sequences for transformer input. A BART encoder-decoder with LoRA adaptation is trained to jointly predict numeric grades and grade buckets, augmented with a distribution-matching term to align predicted and empirical grade distributions, an evaluation dimension often overlooked in prior work.

Experiments compare single-task and multitask training, hard one-hot versus fuzzy and boundary-based soft labels, and rubric versus no-rubric conditions, with additional T5 and pairwise-pretrained variants. Results show that multitask BART with boundary-based soft labels and rubric context achieves lower mean absolute error and stronger grade-distribution alignment than single-task, hard-label, or code-only baselines. Fully fine-tuned T5 further improves distributional fidelity, while pairwise pretraining reduces numeric error at the cost of minority-class sensitivity. Collectively, the findings suggest that calibration-aware, rubric-guided training produces more instructor-like grading behavior than accuracy-optimized alternatives.

Artificial Intelligence (cs.AI)

arXiv:2606.03814 [cs.AI]

(or arXiv:2606.03814v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2606.03814 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Full text under the publisher's open licence · 274 words

Read at arXiv ↗Ask about this article

Related coverage

AI Research

A Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models

arXiv · Nov 4, 2026

AI Research

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

arXiv · Nov 17, 2026

AI Research

A Semitechnical Interlude on the Equivalence of Probabilistic and Deterministic Solomonoff Induction

LessWrong · Sep 2, 2026

AI Research

AgentProofBench: Benchmarking Reusable Verifiers for Web Agents

arXiv · Oct 27, 2026

Article info

Source

arXiv

Published

June 3, 2026

Read time

3 min

Category

AI Products
aiqbench
AI & Quantum Intelligence© aiqbench. All rights reserved.

Feeds:RSS·JSON Feed

LanguageEnglish·Românăbeta
NewsAI & TechQuantumInsights
ExploreBenchmarksLearn
CompanyAboutTrustLegal noticeTerms of ServicePrivacy PolicyCopyright noticesData erasureReport content

The aiqbench take, the insights, the daily digest and Ask answers are written by AI models from the sources shown, without editorial review before publication. They may contain errors — check the source. How the model is used →

  • News
  • Insights
  • Ask AI
  • Benchmarks
  • Account