aiqbench analysis
This study reveals that indiscriminate use of LLMs for grading introduces systemic unfairness due to significant inter-model inconsistencies and grading drift driven by interaction history. Educators must adopt specific operational practices, such as human-aligned workflows, to mitigate these disparities rather than relying on simple ensemble methods. The findings underscore the critical need for rigorous validation before deploying generative AI in high-stakes academic assessment environments.
Impacts of Histories and Models on LLM Grading: A Study in Advanced Software Engineering Courses
arXiv:2606.08400v1 Announce Type: cross Abstract: Graduate-level research reading report assessment creates a substantial labor burden for educators. While large language models (LLMs) hold great potential for automating academic grading, their reliability for this specialized task remains understudied, particularly regarding grading consistency, the lack of which represents a primary obstacle to educational fairness. This paper proposes a human-aligned LLM-assisted grading workflow and presents a case study based on 180 student submissions from a graduate advanced software engineering course. We evaluate two mainstream LLMs, Grok and GPT, in terms of grading consistency and alignment with human scores. We find LLMs exhibit distinct levels of intra-model consistency and significant inter-model grading inconsistencies, while simple ensemble approaches cannot improve alignment with human evaluation. Critically, continuous interaction history drives systematic drift in models' grading standards away from human expert scores. Our findings demonstrate LLMs' potential in reducing grading workload for educators in graduate education, while highlighting that indiscriminate LLM grading may introduce systemic unfairness, suggesting that specific operational practices are required to mitigate such disparities.
Abstract:Graduate-level research reading report assessment creates a substantial labor burden for educators. While large language models (LLMs) hold great potential for automating academic grading, their reliability for this specialized task remains understudied, particularly regarding grading consistency, the lack of which represents a primary obstacle to educational fairness. This paper proposes a human-aligned LLM-assisted grading workflow and presents a case study based on 180 student submissions from a graduate advanced software engineering course. We evaluate two mainstream LLMs, Grok and GPT, in terms of grading consistency and alignment with human scores. We find LLMs exhibit distinct levels of intra-model consistency and significant inter-model grading inconsistencies, while simple ensemble approaches cannot improve alignment with human evaluation. Critically, continuous interaction history drives systematic drift in models' grading standards away from human expert scores. Our findings demonstrate LLMs' potential in reducing grading workload for educators in graduate education, while highlighting that indiscriminate LLM grading may introduce systemic unfairness, suggesting that specific operational practices are required to mitigate such disparities.
Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2606.08400 [cs.SE]
(or arXiv:2606.08400v1 [cs.SE] for this version)
https://doi.org/10.48550/arXiv.2606.08400 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 266 words
Related coverage
Scaling security reviews at 1Password: Building an AI-powered pipeline
1Password · Jun 29, 2026
IEEE’s Global Museum Brings Engineering History to You
IEEE Spectrum · Jul 3, 2026
Why NASA Competes JPL After Caltech’s 1930s Management
Quantum Zeitgeist · May 25, 2026
Caltech Loses Sole-Source JPL Contract After Decades
Quantum Zeitgeist · May 25, 2026