analiza aiqbench
This study reveals that current LLMs struggle to accurately predict human item discrimination, a critical psychometric property essential for distinguishing between students of varying proficiency levels in reading comprehension assessments. While response-based proxy methods offer some ranking signal, the findings indicate that models have not yet mastered simulating ability-conditioned human behavior required for robust educational evaluation metrics. For developers focusing on AI-driven assessment tools, this highlights a significant gap between current generative capabilities and reliable psychometric modeling.
LLMs Struggle to Measure What Distinguishes Students of Different Proficiency Levels: A Study of Item Discrimination in Reading Comprehension Assessment
arXiv:2606.18709v2 Announce Type: replace Abstract: Existing work on LLM-based educational assessment has focused largely on item difficulty, but difficulty alone does not indicate whether an item meaningfully distinguishes higher- from lower-proficiency students. Item discrimination captures this complementary and fundamental psychometric property. We investigate whether LLMs can predict human item discrimination from assessment content. We evaluate 42 proprietary and open-weight LLMs using two complementary approaches. Direct discrimination prediction asks models to explicitly predict an item's discrimination value, while response-based proxy estimation treats LLM answers as synthetic responses and applies a Classical Test Theory (CTT)-inspired item-rest calculation. Direct predictions show weak alignment with human item discrimination. The response-based proxy provides a stronger but still limited ranking signal, reaching a CEFR-stratified rank correlation of 0.231. Further analysis shows that this correlation comes mainly from differences across models rather than proficiency prompts that reliably simulate students at different ability levels. Current LLMs therefore contain some discrimination-relevant information, but they do not yet reliably model the ability-conditioned human response behavior that gives item discrimination its psychometric meaning.
Abstract:Existing work on LLM-based educational assessment has focused largely on item difficulty, but difficulty alone does not indicate whether an item meaningfully distinguishes higher- from lower-proficiency students. Item discrimination captures this complementary and fundamental psychometric property. We investigate whether LLMs can predict human item discrimination from assessment content. We evaluate 42 proprietary and open-weight LLMs using two complementary approaches. Direct discrimination prediction asks models to explicitly predict an item's discrimination value, while response-based proxy estimation treats LLM answers as synthetic responses and applies a Classical Test Theory (CTT)-inspired item-rest calculation. Direct predictions show weak alignment with human item discrimination. The response-based proxy provides a stronger but still limited ranking signal, reaching a CEFR-stratified rank correlation of 0.231. Further analysis shows that this correlation comes mainly from differences across models rather than proficiency prompts that reliably simulate students at different ability levels. Current LLMs therefore contain some discrimination-relevant information, but they do not yet reliably model the ability-conditioned human response behavior that gives item discrimination its psychometric meaning.
Computation and Language (cs.CL)
arXiv:2606.18709 [cs.CL]
(or arXiv:2606.18709v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.18709 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 260 cuvinte
Subiecte conexe
Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
LessWrong · 5 oct. 2026
SWIM: Student Writing Simulation via Proficiency-Conditioned Generation
arXiv · 4 sept. 2026
Frontier AI Risk Report (February–March 2026)
METR · 19 mai 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · 19 mai 2026