aiqbench analysis
This study reveals significant unreliability in AI grading systems, which can deviate from human assessments by up to 40 points due to an inability to evaluate nuanced argumentation and context-specific errors. The findings suggest that while well-defined rubrics can improve alignment, current generative models lack the transparency and depth required for high-stakes educational evaluation without rigorous human oversight.
AI grading can differ from a human’s by 40 points, study finds
A recent study found AI-generated marks varied from lecturer assessments by as much as 40 points, raising questions about justification and AI detection accuracy.
A lecturer’s grade and an AI’s assessment of the same student work differed by as many as 40 points out of 100, according to recent findings that raise questions about the reliability of artificial intelligence in education. Researchers analyzed 85 studies, including a meta-analysis of 49 studies comprising 59 effects, to examine AI’s impact on STEM learning, but ultimately concluded that a reliable overall effect of generative AI use on learning could not be confirmed. The team also discovered hidden grading instructions within the initial feedback generated by an AI system, highlighting a lack of transparency, Meta-Analysis says. “Checking and explaining AI-generated responses, using them transparently and taking responsibility for them are essential,” the Center for the use of ICT in pedagogical process reports. Assessment scores produced by artificial intelligence deviated from those assigned by human lecturers by as much as 40 points on a 100-point scale, according to a systematic review of 85 studies. This discrepancy highlights a critical unreliability in…
Excerpt shown under the publisher's RSS terms · 162 words · full text and figures are on the publisher's site
