aiqbench analysis
This study provides critical insight into how Large Language Models are evaluated within educational contexts, revealing that LLMs significantly outperform students on structural and orthographic metrics while humans retain an edge in creativity and natural phrasing. The research highlights the limitations of aggregate scoring by demonstrating that granular criterion-level auditing is necessary to understand where AI excels versus where it lacks human nuance. For educators and researchers assessing generative tools, this underscores the importance of dissecting specific rubric components rather than relying on total scores alone.
When LLM Essays Outscore Student Essays: What a Korean Writing Rubric Rewards and Where Readers Disagree
arXiv:2601.19913v4 Announce Type: replace-cross Abstract: LLMs now help students plan, draft, and revise essays. Educational assessment therefore faces a basic question: how should student and LLM writing be compared? Rubrics assign points to content, organization, and expression. Their total can still hide which criteria drive the comparison, where ratings approach the maximum, and where readers disagree. We therefore conducted a secondary, post hoc audit of a Korean writing study with source-informed scoring. Three Korean language and literature majors used a 16-criterion, 100-point rubric to score six student essays and 24 essays from four LLMs prompted for three school levels. After source disclosure, they helped develop the rubric and, according to the protocol, scored the Phase 2 essays without gold labels while recording source judgments. The pooled LLM mean exceeds the student mean by 18.35 points. Orthographic norms and genre-appropriate register contribute 8.19 points, or 44.7% of the gap. All 72 LLM ratings reach the orthographic maximum, and 70 reach the register maximum. The student source mean is highest or tied highest on both creativity criteria and natural Korean phrasing, where reader agreement is weak. Criterion-level auditing therefore offers a deeper account of human and LLM writing than the total alone.
Abstract:LLMs now help students plan, draft, and revise essays. Educational assessment therefore faces a basic question: how should student and LLM writing be compared? Rubrics assign points to content, organization, and expression. Their total can still hide which criteria drive the comparison, where ratings approach the maximum, and where readers disagree. We therefore conducted a secondary, post hoc audit of a Korean writing study with source-informed scoring. Three Korean language and literature majors used a 16-criterion, 100-point rubric to score six student essays and 24 essays from four LLMs prompted for three school levels. After source disclosure, they helped develop the rubric and, according to the protocol, scored the Phase 2 essays without gold labels while recording source judgments. The pooled LLM mean exceeds the student mean by 18.35 points. Orthographic norms and genre-appropriate register contribute 8.19 points, or 44.7% of the gap. All 72 LLM ratings reach the orthographic maximum, and 70 reach the register maximum. The student source mean is highest or tied highest on both creativity criteria and natural Korean phrasing, where reader agreement is weak. Criterion-level auditing therefore offers a deeper account of human and LLM writing than the total alone.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2601.19913 [cs.CL]
(or arXiv:2601.19913v4 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2601.19913 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 291 words
Related coverage
Daniel Ritter
file770.com · Aug 29, 2026
Links #5: 2026/07
LessWrong · Jul 31, 2026
SWIM: Student Writing Simulation via Proficiency-Conditioned Generation
arXiv · Sep 4, 2026
Contaminated Collaboration: Measuring Gender Bias Transfer in LLM-Assisted Student Writing
arXiv · Jun 17, 2026