Detecting Knowledge Gaps from Conversational AI Interactions Using Curriculum Prerequisite Graphs

Abstract:Large online courses generate thousands of student questions directed at conversational AI teaching assistants, yet these interaction logs remain largely untapped as diagnostic signals. We present a pipeline that maps student questions from a conversational AI teaching assistant to curriculum topics using a few-shot text classifier, grounded in a GPT-4-extracted prerequisite knowledge graph of course concepts. Evaluated on 1,340 question events from 164 students in a graduate-level AI course, our classifier achieves 80.0% accuracy across 43 labels (42 curriculum topics plus an "unknown" abstention class). Topic-level question volume correlates significantly with student self-reported difficulty from an independent mid-semester survey (rho = 0.491, p = 0.008, n = 28 topics), providing convergent evidence that the classified question stream reflects genuine topic difficulty. These results demonstrate that conversational AI interaction logs, mapped onto curriculum structure, carry actionable signals about topic-level knowledge gaps and provide instructors with a curriculum-grounded view of which topics warrant attention.

Accepted as a short paper at the 10th CSEDM Workshop, co-located with the 18th International Conference on Educational Data Mining (EDM 2026). 7 pages, 2 figures, 2 tables

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)

I.2.4; I.2.6; I.2.7; K.3.1

arXiv:2606.10736 [cs.CL]

(or arXiv:2606.10736v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2606.10736 arXiv-issued DOI via DataCite (pending registration)