aiqbench analysis
This arXiv study tests whether small, locally run language models and a vision-language model can grade student incident-response reports against rubrics. It reports up to 97% accuracy on text-only reports and 94% on rich-text reports, and says this is comparable to a large model. The results come from 105 students over two semesters, so they are promising but limited to one course and one task. For AI readers, the main takeaway is the trade-off between privacy, cost, and hardware constraints on one side and accuracy on the other.
Rubric-Based Assessment of Student Reports Using Small Language Models
arXiv:2611.01785v1 Announce Type: cross Abstract: Language models can analyze and assess student texts using predefined rubrics and deliver structured feedback, reducing instructors' workload. However, existing solutions often rely on external services that are resource-intensive, costly, and raise privacy concerns, making them unsuitable for constrained environments. This study evaluates an automated assessment pipeline with three key aims: design for low-resource environments, privacy, and focus on a novel context in cybersecurity education. The pipeline integrates lightweight language models for textual evaluation and a vision-language model for image interpretation. Responses from 105 students were collected over two semesters of an undergraduate course during a cybersecurity exercise on incident response. Results across different hardware configurations show that small models in low-resource environments can achieve up to 97% accuracy in text-only and up to 94% in rich-text assessments. This rivals or even outperforms a large model while adding the benefits of locally hosted assessment. Our solution can help educators and practitioners who need grading assistance without replacing human oversight, especially in environments where cloud-based solutions may be impractical. This study contributes an open-source, locally deployable AI-assisted scoring pipeline that runs on affordable hardware and can be extended to various types of student reports in computing education. As a result, it makes automated assessment accessible to a broad range of educators and institutions.
Abstract:Language models can analyze and assess student texts using predefined rubrics and deliver structured feedback, reducing instructors' workload. However, existing solutions often rely on external services that are resource-intensive, costly, and raise privacy concerns, making them unsuitable for constrained environments. This study evaluates an automated assessment pipeline with three key aims: design for low-resource environments, privacy, and focus on a novel context in cybersecurity education. The pipeline integrates lightweight language models for textual evaluation and a vision-language model for image interpretation. Responses from 105 students were collected over two semesters of an undergraduate course during a cybersecurity exercise on incident response. Results across different hardware configurations show that small models in low-resource environments can achieve up to 97% accuracy in text-only and up to 94% in rich-text assessments. This rivals or even outperforms a large model while adding the benefits of locally hosted assessment. Our solution can help educators and practitioners who need grading assistance without replacing human oversight, especially in environments where cloud-based solutions may be impractical. This study contributes an open-source, locally deployable AI-assisted scoring pipeline that runs on affordable hardware and can be extended to various types of student reports in computing education. As a result, it makes automated assessment accessible to a broad range of educators and institutions.
Computers and Society (cs.CY); Computation and Language (cs.CL); Cryptography and Security (cs.CR)
arXiv:2611.01785 [cs.CY]
(or arXiv:2611.01785v1 [cs.CY] for this version)
https://doi.org/10.48550/arXiv.2611.01785 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
https://doi.org/10.1145/3857222.3860169 Focus to learn more DOI(s) linking to related resources
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 325 words