aiqbench analysis
The paper argues that current temporal knowledge graph forecasting benchmarks can be gamed: for some test queries, models score well without using any time information, because dataset biases and a simplistic task setup let them exploit shortcuts. Its proposed benchmark, TKGB, adds four tasks, up-to-date and semantically enriched data, and controls for shortcuts. For ML practitioners, the main takeaway is that leaderboard gains on existing TKG datasets may not reflect real temporal reasoning, so evaluation design deserves as much scrutiny as model architecture.
TKGB: A Comprehensive Benchmark for Temporal Knowledge Graph Forecasting
arXiv:2602.08353v2 Announce Type: replace Abstract: Temporal knowledge graph (TKG) forecasting has advanced remarkably in recent years. Yet we found existing benchmarks inadvertently introduce a shortcut: For a subset of test queries, models can achieve optimal evaluation metrics without using any time information. We trace this to two pieces of evidence: inherent dataset biases and an overly simplistic task formulation that allows models to exploit biases, revealing fundamental flaws in current evaluation protocols for differentiating true forecasting ability. Moreover, existing datasets rely on fixed history that poorly reflects real-world dynamics, lack semantic context, and fail to explicitly mitigate shortcut, preventing thorough assessment of forecasting abilities. To address these, we propose TKGB, a comprehensive benchmark for evaluating the capability of TKG forecasting models. By integrating four complementary tasks with an automatic pipeline for constructing up-to-date and context-enriched TKG datasets with controllable shortcuts, TKGB reveals capability differences and failure modes hidden under conventional evaluation.
Abstract:Temporal knowledge graph (TKG) forecasting has advanced remarkably in recent years. Yet we found existing benchmarks inadvertently introduce a shortcut: For a subset of test queries, models can achieve optimal evaluation metrics without using any time information. We trace this to two pieces of evidence: inherent dataset biases and an overly simplistic task formulation that allows models to exploit biases, revealing fundamental flaws in current evaluation protocols for differentiating true forecasting ability. Moreover, existing datasets rely on fixed history that poorly reflects real-world dynamics, lack semantic context, and fail to explicitly mitigate shortcut, preventing thorough assessment of forecasting abilities. To address these, we propose TKGB, a comprehensive benchmark for evaluating the capability of TKG forecasting models. By integrating four complementary tasks with an automatic pipeline for constructing up-to-date and context-enriched TKG datasets with controllable shortcuts, TKGB reveals capability differences and failure modes hidden under conventional evaluation.
Artificial Intelligence (cs.AI)
arXiv:2602.08353 [cs.AI]
(or arXiv:2602.08353v2 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2602.08353 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 238 words
Related coverage
Chronofy: A Temporal-Logical Decay Architecture for Information Validity in Time-Aware Retrieval-Augmented Generation
arXiv · Jul 24, 2026
Graph-Based Modeling of Financial Volatility Dynamics
arXiv · Aug 28, 2026
SpikF-GO: Spiking Fourier Graph Operators for Multivariate Time Series Forecasting
arXiv · Jun 15, 2026
A Temporal Knowledge Graph for Music Festival Lineup Forecasting
arXiv · Sep 22, 2026