aiqbench analysis
This research introduces GAOKAO-Bench, a rigorous evaluation framework using Chinese high-stakes exam questions to assess Large Language Models in zero-shot settings. The study reveals significant performance disparities across subjects and demonstrates that while models can grade subjective questions with moderate consistency to human evaluators, they still face limitations in specific domains. This benchmark provides essential data for understanding the current capabilities and boundaries of LLMs in complex, culturally specific reasoning tasks.
Evaluating the Performance of Large Language Models on GAOKAO Benchmark
arXiv:2305.12474v4 Announce Type: replace-cross Abstract: Large Language Models(LLMs) have demonstrated remarkable performance across various natural language processing tasks; however, how to comprehensively and accurately assess their performance becomes an urgent issue to be addressed. This paper introduces GAOKAO-Bench, an intuitive benchmark that employs questions from the Chinese GAOKAO examination as test samples, including both subjective and objective questions. To align with human examination methods, we design a method based on zero-shot settings to evaluate the performance of LLMs. With human evaluation, we obtain the converted total score of LLMs, including GPT-4, ChatGPT and ERNIE-Bot.Our findings reveal that LLMs have achieved competitive scores in Chinese GAOKAO examination, while they exhibit significant performance disparities across various subjects. We also use LLMs to grade the subjective questions, and find that model scores achieve a moderate level of consistency with human scores. In conclusion, this research contributes a robust evaluation benchmark for future large language models and offers valuable insights into the advantages and limitations of such models.
Abstract:Large Language Models(LLMs) have demonstrated remarkable performance across various natural language processing tasks; however, how to comprehensively and accurately assess their performance becomes an urgent issue to be addressed. This paper introduces GAOKAO-Bench, an intuitive benchmark that employs questions from the Chinese GAOKAO examination as test samples, including both subjective and objective questions. To align with human examination methods, we design a method based on zero-shot settings to evaluate the performance of LLMs. With human evaluation, we obtain the converted total score of LLMs, including GPT-4, ChatGPT and this http URL findings reveal that LLMs have achieved competitive scores in Chinese GAOKAO examination, while they exhibit significant performance disparities across various subjects. We also use LLMs to grade the subjective questions, and find that model scores achieve a moderate level of consistency with human scores. In conclusion, this research contributes a robust evaluation benchmark for future large language models and offers valuable insights into the advantages and limitations of such models.
Updated the author metadata to match the manuscript and added Qixiang Wang in recognition of his contribution to data collection and curation. Results and conclusions are unchanged
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2305.12474 [cs.CL]
(or arXiv:2305.12474v4 [cs.CL] for this version)
Full text under the publisher's open licence · 203 words
Related coverage
MedPAIR: Measuring Whether Physicians and AI Agree on What Matters in Medical QA
Snorkel AI Blog · Oct 2, 2026
[AINews] not much happened today
Latent Space · Sep 10, 2026
How to Evaluate General-Purpose Robot Policies for Real-World Deployment
NVIDIA · Jul 12, 2026
Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model
AMD · Jul 24, 2026