Evaluation Choices Decide the Forecasting Leaderboard: Evidence from a Production Marketplace Panel

Abstract:A forecasting benchmark reports which method won. We show that the answer is set by the evaluator's choices before any model is fitted. We benchmark 24 forecasting methods and one textbook reference, including six 2025-era time series foundation models, on a production marketplace panel of 1,887 business customers over 67 months. We hold the data, the horizon and the period fixed, and vary only the evaluation design. Three choices each reverse or dissolve a headline conclusion. Changing the unit of analysis from the market total to the individual customer moves our production baseline from second of nineteen, beaten by nothing, to twenty-third of twenty-five. Nineteen of its twenty-four challengers beat it there. Changing how much error is pooled decides whether a Diebold-Mariano test finds anything at all. Scoring prediction intervals rather than point forecasts reorders the field almost completely, with a rank correlation of 0.02 on intermittent demand. We then measure what the deployed system gets from this. Its selection rule captures 55% of the distance between doing nothing and choosing with hindsight. The reversal is not a quirk of our data. We ran the released protocol, unchanged, on the public M5 retail panel. The same baseline shape places first at the market total and last per series, beaten by everything, and a replayed selection rule closes 64.7% of the same floor-to-ceiling distance there. Adding five zero-shot foundation models to that roster changes who wins at the total, not the shape. The bands' blind spot travels too: conformal bands under-cover most on the spikiest items. Splitting our own panel into ever smaller groups turns the contrast into a curve: the baseline's rank worsens at every level of disaggregation. We release the evaluation protocol and report an error of our own that inverted a result before we caught it.

18 pages, 8 tables, 1 figure. Evaluation protocol and audit scripts included as ancillary files

Machine Learning (cs.LG); Applications (stat.AP); Methodology (stat.ME)

arXiv:2609.27867 [cs.LG]

(or arXiv:2609.27867v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2609.27867 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

p6_evaluation_protocol/PROTOCOL.md

p6_evaluation_protocol/README.md

p6_evaluation_protocol/audit/README.md

p6_evaluation_protocol/audit/_provenance.py

p6_evaluation_protocol/audit/audit_selection_rule_and_corrections.py

p6_evaluation_protocol/audit/backtest_per_buyer_multi_origin.py

p6_evaluation_protocol/audit/build_exhibits.py

p6_evaluation_protocol/audit/extend_m5_roster_with_foundation_models.py

p6_evaluation_protocol/audit/probe_aggregation_cancellation.py

p6_evaluation_protocol/audit/probe_leaderboard_rank_bootstrap.py

p6_evaluation_protocol/audit/probe_one_champion_policy.py

p6_evaluation_protocol/audit/replicate_inversion_on_m5_public.py

p6_evaluation_protocol/audit/replicate_inversion_on_tourism_public.py

p6_evaluation_protocol/audit/score_m5_extended_roster_intervals.py

p6_evaluation_protocol/audit/score_m5_interval_quality.py

p6_evaluation_protocol/audit/score_rolling_interval_calibration.py

p6_evaluation_protocol/audit/sweep_aggregation_dose_response.py

p6_evaluation_protocol/audit/sweep_m5_native_hierarchy_levels.py

p6_evaluation_protocol/audit/sweep_synthetic_smoothness_dial.py

p6_evaluation_protocol/metrics.py

p6_evaluation_protocol/protocol_harness.py

p6_evaluation_protocol/pyproject.toml

(17 additional files not shown) You must enabled JavaScript to view entire file list.

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.