SDUs DAISY: A Benchmark for Danish Culture

arXiv:2601.19930v2 Announce Type: replace-cross Abstract: We introduce Daisy, a factual knowledge benchmark for Danish cultural heritage, based on curated topics from the Danish Culture Canon 2006. For each artifact in the culture canon, we query the corresponding Wikipedia page and have a language model generate a diverse set of questions. Within each artifact, we sample both central and peripheral questions, testing not only mainstream information but also the deeper, defining elements of Danish cultural heritage as identified by the Canon committee. Each question-answer pair is manually approved or corrected, yielding a final dataset of 741 closed-ended question-answer pairs ranging from archaeological findings dated to 1300 BCE and 18th-century poems and musical pieces through to contemporary pop music, Danish design, and architecture. Baseline results on our benchmark show that contemporary language models (GPT-OSS-120B & 20B, Llama-3.3-70B, Gemma3-27B, and Mistral-3.1-24B) perform surprisingly poorly on this task, despite the tested factual knowledge being available on Wikipedia. The best-performing model, Llama-3.3-70B, attains only 0.17 BLEU and an F1 score of 0.27, demonstrating the benchmark's difficulty. The questions, model benchmarks, and the evaluation tool are publicly available at https://github.com/schneiderkamplab/SDU-Daisy and under schneiderkamplab/SDU-Daisy on Hugging Face Datasets.

Abstract:We introduce Daisy, a factual knowledge benchmark for Danish cultural heritage, based on curated topics from the Danish Culture Canon 2006. For each artifact in the culture canon, we query the corresponding Wikipedia page and have a language model generate a diverse set of questions. Within each artifact, we sample both central and peripheral questions, testing not only mainstream information but also the deeper, defining elements of Danish cultural heritage as identified by the Canon committee. Each question-answer pair is manually approved or corrected, yielding a final dataset of 741 closed-ended question-answer pairs ranging from archaeological findings dated to 1300 BCE and 18th-century poems and musical pieces through to contemporary pop music, Danish design, and architecture. Baseline results on our benchmark show that contemporary language models (GPT-OSS-120B & 20B, Llama-3.3-70B, Gemma3-27B, and Mistral-3.1-24B) perform surprisingly poorly on this task, despite the tested factual knowledge being available on Wikipedia. The best-performing model, Llama-3.3-70B, attains only 0.17 BLEU and an F1 score of 0.27, demonstrating the benchmark's difficulty. The questions, model benchmarks, and the evaluation tool are publicly available at this https URL and under schneiderkamplab/SDU-Daisy on Hugging Face Datasets.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

arXiv:2601.19930 [cs.CL]

(or arXiv:2601.19930v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2601.19930 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

SDUs DAISY: A Benchmark for Danish Culture

Subiecte conexe