aiqbench analysis
This study introduces the first benchmark evaluating large language models on Polish high school history exams, revealing critical failure modes like source decontextualization and temporal disorientation despite near-ceiling aggregate scores. The research highlights that model performance varies significantly based on task type and geographical scope, with a consistent penalty for local versus global historical content. These findings underscore the limitations of current LLMs in handling complex interpretative reasoning required by specific national curricula rather than general knowledge retrieval.
Lost in Historical Time? A Polish History Matura Benchmark for Large Language Models
arXiv:2608.12343v2 Announce Type: replace Abstract: Language models are widely used by students as knowledge sources, yet benchmarks rarely assess their interpretative historical reasoning. We evaluate eight leading LLMs on the Polish high school exit exam (Matura) in history - three official papers from 2023-2025, comprising short-answer questions and extended essays - and compare model performance against the human examinee population. Although models score near the ceiling, aggregate scores mask distinct competency profiles: rankings are unstable across task types, source modalities, and geographical scopes, with a consistent penalty for Polish versus Global history content. Qualitative error analysis reveals two recurring failure modes - source decontextualization, when models reason from source content rather than treating it as an object of analysis, and temporal disorientation, when responses are historically misplaced. This study introduces the first LLM history benchmark grounded in the Polish national curriculum.
Title:Lost in Historical Time? A Polish History Matura Benchmark for Large Language Models
Abstract:Language models are widely used by students as knowledge sources, yet benchmarks rarely assess their interpretative historical reasoning. We evaluate eight leading LLMs on the Polish high school exit exam (Matura) in history - three official papers from 2023-2025, comprising short-answer questions and extended essays - and compare model performance against the human examinee population. Although models score near the ceiling, aggregate scores mask distinct competency profiles: rankings are unstable across task types, source modalities, and geographical scopes, with a consistent penalty for Polish versus Global history content. Qualitative error analysis reveals two recurring failure modes - source decontextualization, when models reason from source content rather than treating it as an object of analysis, and temporal disorientation, when responses are historically misplaced. This study introduces the first LLM history benchmark grounded in the Polish national curriculum.
Computation and Language (cs.CL)
arXiv:2608.12343 [cs.CL]
(or arXiv:2608.12343v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.12343 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 242 words
