aiqbench analysis
This research demonstrates how fine-tuned Large Language Models can simulate student responses to generate synthetic Item Characteristic Curves, offering a cost-effective alternative to traditional field testing for psychometric calibration. By leveraging Qwen-3 with LoRA, the study effectively reconstructs probability functions based on latent ability descriptors without requiring extensive real-world data collection. The approach proves particularly effective at modeling item discrimination in educational assessment contexts like Grade 6 ELA and BEA shared tasks.
Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models
arXiv:2601.02580v2 Announce Type: replace-cross Abstract: Traditional methods for determining assessment item parameters, such as difficulty and discrimination, rely heavily on expensive field testing to collect student performance data for Item Response Theory (IRT) calibration. This study introduces a novel approach that implicitly models these psychometric properties by fine-tuning Large Language Models (LLMs) to simulate student responses across a spectrum of latent abilities. Leveraging the Qwen-3 dense model series and Low-Rank Adaptation (LoRA), we train models to generate responses to multiple choice questions conditioned on discrete ability descriptors. We reconstruct the probability of a correct response as a function of student ability, effectively generating synthetic Item Characteristic Curves (ICCs) to estimate IRT parameters. Evaluation on a dataset of Grade 6 English Language Arts (ELA) items and the BEA 2024 Shared Task dataset demonstrates that this method competes with or outperforms baseline approaches. This simulation-based technique seems particularly effective at modeling item discrimination.
Abstract:Traditional methods for determining assessment item parameters, such as difficulty and discrimination, rely heavily on expensive field testing to collect student performance data for Item Response Theory (IRT) calibration. This study introduces a novel approach that implicitly models these psychometric properties by fine-tuning Large Language Models (LLMs) to simulate student responses across a spectrum of latent abilities. Leveraging the Qwen-3 dense model series and Low-Rank Adaptation (LoRA), we train models to generate responses to multiple choice questions conditioned on discrete ability descriptors. We reconstruct the probability of a correct response as a function of student ability, effectively generating synthetic Item Characteristic Curves (ICCs) to estimate IRT parameters. Evaluation on a dataset of Grade 6 English Language Arts (ELA) items and the BEA 2024 Shared Task dataset demonstrates that this method competes with or outperforms baseline approaches. This simulation-based technique seems particularly effective at modeling item discrimination.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2601.02580 [cs.CL]
(or arXiv:2601.02580v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2601.02580 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 242 words
Related coverage
A Self-Supervised Framework for Space Object Behaviour Characterisation
arXiv · Jul 23, 2026
Links #3: 2026/06 Part 1
LessWrong · Jun 15, 2026
Building Blocks for Foundation Model Training and Inference on AWS
Hugging Face · May 11, 2026
[AINews] not much happened today
Latent Space · Oct 3, 2026