analiza aiqbench
This research exposes a critical reproducibility flaw in LLM evaluation where hidden date injections in system prompts cause performance deltas of up to 14% on math reasoning tasks, surpassing variations from hardware or batching. The study demonstrates that standard prompting techniques like chain-of-thought fail to mitigate this sensitivity and may even amplify it, directly impacting the fairness of model leaderboards. Consequently, the field requires updated evaluation protocols that account for temporal dependencies to ensure robust and comparable benchmark results.
Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation
arXiv:2609.36931v1 Announce Type: new Abstract: Reproducibility is essential for scientific research, yet prior work shows that LLM outputs vary with hardware and batching. We identify an overlooked factor: the hidden injection of the current date into system prompts, which users cannot control and which changes every day. Across 9 recent LLMs and 6 datasets spanning multiple-choice QA (MCQA), math reasoning, code generation, and machine translation, performance varies solely with the current date, with deltas of up to 6% on MCQA, 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation. Model rankings also shift, affecting leaderboards. This date effect exceeds other sources of non-determinism, such as batch size and numerical precision. Standard prompting techniques -- chain-of-thought and few-shot prompting -- do not reduce the sensitivity; chain-of-thought even amplifies it. Our findings underscore the need for careful evaluation protocols to ensure reproducibility and fair comparisons in LLM research.
Abstract:Reproducibility is essential for scientific research, yet prior work shows that LLM outputs vary with hardware and batching. We identify an overlooked factor: the hidden injection of the current date into system prompts, which users cannot control and which changes every day. Across 9 recent LLMs and 6 datasets spanning multiple-choice QA (MCQA), math reasoning, code generation, and machine translation, performance varies solely with the current date, with deltas of up to 6% on MCQA, 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation. Model rankings also shift, affecting leaderboards. This date effect exceeds other sources of non-determinism, such as batch size and numerical precision. Standard prompting techniques -- chain-of-thought and few-shot prompting -- do not reduce the sensitivity; chain-of-thought even amplifies it. Our findings underscore the need for careful evaluation protocols to ensure reproducibility and fair comparisons in LLM research.
Computation and Language (cs.CL)
arXiv:2609.36931 [cs.CL]
(or arXiv:2609.36931v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.36931 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 241 cuvinte
Subiecte conexe
Hint-based CoT faithfulness evals still mostly work on Claude
LessWrong · 30 iul. 2026
Investigating three real-world incidents in our cybersecurity evaluations
Anthropic · 30 iul. 2026
Evaluating task vectors, unlearning and inoculation
LessWrong · 20 sept. 2026
The current bottleneck is political will, not research
AI Alignment Forum · 11 iul. 2026