A Large-Scale Chinese Knowledge Graph-Text Alignment Dataset for Benchmarking Knowledge-Grounded LLMs

Abstract:Reliable evaluation of knowledge-grounded Large Language Models (LLMs) in Chinese requires resources that explicitly align Chinese-language text with verifiable Knowledge Graph (KG) facts. Yet existing Chinese benchmarks primarily assess general language understanding and offer limited support for structured reasoning under Chinese-specific linguistic phenomena. We introduce the Chinese Data-Text Pair (CDTP), a large-scale Chinese KG-text alignment dataset comprising more than 7 million aligned instances across four broad domains. Each instance pairs a Chinese-language text with one or more textually supported KG triples, totaling 15 million triples. A multi-stage construction pipeline combining alignment filtering, manual verification, and external evidence validation improves semantic consistency and factual reliability. CDTP supports Knowledge Graph Completion (KGC), Question Answering (QA), and Triple-to-Text Generation (T2T). Across all three tasks, the benchmark design accounts for Chinese-specific phenomena, including polysemy, word-segmentation ambiguity, and context-dependent entity interpretation, enabling the evaluation of structured reasoning, ambiguity-aware factual understanding, and knowledge-grounded generation. Experiments with diverse open-source and proprietary LLMs show that model scale alone does not guarantee reliable performance on these Chinese knowledge-intensive tasks, whereas supervised fine-tuning on CDTP consistently improves in-domain performance and out-of-distribution robustness. The publicly accessible dataset, code, and evaluation protocols provide a reusable resource for developing and evaluating knowledge-grounded LLMs in Chinese.

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

arXiv:2510.06039 [cs.CL]

(or arXiv:2510.06039v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2510.06039 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.