aiqbench analysis
AfriEconQA introduces a rigorous benchmark for document-grounded question answering, specifically testing large language models on their ability to extract precise numerical and temporal details from dense World Bank economic reports. By evaluating leading open-source models like Qwen 3.6 and Gemma 4 under zero-shot and RAG conditions, the study highlights persistent challenges in maintaining factual accuracy across complex reasoning tasks involving cross-period comparisons. This work establishes a critical new standard for advancing AI systems capable of reliable quantitative analysis over long institutional documents.
AfriEconQA: A Benchmark for Quantitative and Temporal Reasoning over World Bank Economic Reports
arXiv:2601.15297v3 Announce Type: replace Abstract: Reliable question answering over long institutional documents requires more than topical retrieval: a system must localize the exact passage that supports a claim and preserve precise numerical and temporal detail when the same indicator recurs across years, countries, and projection horizons in dense, repetitive prose. Existing question-answering benchmarks rarely test this combination of exact grounding and temporal precision at document scale. We introduce AfriEconQA, a benchmark for document-grounded question answering built from 220 World Bank economic reports on African economies, a corpus whose claims are tied to specific fiscal periods, countries, and projection states. AfriEconQA contains 4,309 evidence-linked QA instances across five reasoning categories: Factoid (1,093), List (879), Multiple Choice (944), Synthesis (710), and Comparison (683), targeting quantitative extraction, set recovery, discrimination, causal integration, and cross-period or cross-country reasoning. Each instance carries sup- porting evidence and source provenance and is constructed through an agentic generation pipeline with evidence-grounding checks and a stratified human-validation subset for gold-label auditing. We evaluate Qwen 3.6 35B, DeepSeek v4-pro, and Gemma 4 12B IT under zero-shot, oracle, and hybrid retrieval-augmented generation (RAG) conditions on the held-out test split (n = 862). Across all three models, retrieval yields substantial gains, yet the best RAG system reaches only 0.545 F1, with residual errors concentrated in list extraction, synthesis, and temporally scoped comparison. AfriEconQA therefore poses a hard open challenge for exact numerical and temporal grounding over long institutional economic reports.
Abstract:Reliable question answering over long institutional documents requires more than topical retrieval: a system must localize the exact passage that supports a claim and preserve precise numerical and temporal detail when the same indicator recurs across years, countries, and projection horizons in dense, repetitive prose. Existing question-answering benchmarks rarely test this combination of exact grounding and temporal precision at document scale. We introduce AfriEconQA, a benchmark for document-grounded question answering built from 220 World Bank economic reports on African economies, a corpus whose claims are tied to specific fiscal periods, countries, and projection states. AfriEconQA contains 4,309 evidence-linked QA instances across five reasoning categories: Factoid (1,093), List (879), Multiple Choice (944), Synthesis (710), and Comparison (683), targeting quantitative extraction, set recovery, discrimination, causal integration, and cross-period or cross-country reasoning. Each instance carries sup- porting evidence and source provenance and is constructed through an agentic generation pipeline with evidence-grounding checks and a stratified human-validation subset for gold-label auditing. We evaluate Qwen 3.6 35B, DeepSeek v4-pro, and Gemma 4 12B IT under zero-shot, oracle, and hybrid retrieval-augmented generation (RAG) conditions on the held-out test split (n = 862). Across all three models, retrieval yields substantial gains, yet the best RAG system reaches only 0.545 F1, with residual errors concentrated in list extraction, synthesis, and temporally scoped comparison. AfriEconQA therefore poses a hard open challenge for exact numerical and temporal grounding over long institutional economic reports.
Computation and Language (cs.CL)
arXiv:2601.15297 [cs.CL]
(or arXiv:2601.15297v3 [cs.CL] for this version)
Full text under the publisher's open licence · 245 words
Related coverage
[AINews] Opus 5.5 is good at explainer videos
Latent Space · Sep 29, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
METR · Jul 21, 2026
CycliST: A Video Language Model Benchmark for Reasoning on Cyclical State Transitions
arXiv · Jun 16, 2026