aiqbench analysis
This is a controlled ML benchmark, not a trading result. Nine architectures are compared on hourly OHLCV forecasting across 12 assets at 4- and 24-hour horizons, with ModernTCN and PatchTST ranking highest on RMSE. The mean directional accuracy of 50.08% is near chance, so the practical value is in the reproducible evaluation methodology rather than in any claim of profitable prediction. The two abstract versions also report different p-values for the same Iman-Davenport test (p=0.683 vs. p<1e-15), which readers should check against the source paper.
A Controlled Comparison of Deep Learning Architectures for Multi-Horizon Financial Forecasting: Evidence from 918 Experiments
arXiv:2603.16886v2 Announce Type: replace-cross Abstract: Reliable comparison of deep learning architectures for financial forecasting requires controlled evaluation that separates architectural effects from hyperparameter tuning, random initialization, forecast horizon, and asset characteristics. We present a five-stage benchmark using equal-budget Bayesian hyperparameter optimization, category-level configuration freezing, three-seed final training, held-out metric aggregation, and rank-based statistical validation. Nine architectures from Transformer, MLP/linear, convolutional, and recurrent families are evaluated with direct multi-step prediction from five OHLCV features across 12 assets (four cryptocurrency, four foreign exchange, four equity-index instruments), using chronological 70/15/15 splits of 30,000 hourly windows per asset at 4- and 24-hour horizons. The benchmark includes 918 runs (270 optimization trials, 648 final training runs). Across 24 asset-horizon points, ModernTCN has mean rank 1.333 and the lowest RMSE in 18 points (75.0%), followed by PatchTST (mean rank 2.000). Global ranking differences are strongly supported by an Iman-Davenport test (F(8,184)=101.36, p=0.683; Stouffer p=3.47e-10), while errors increase at the longer horizon. Architecture explains 99.90% of raw error variance and 68.33% after within-slot normalization excluding LSTM; seed variance contributes at most 0.04% (0.02% without LSTM). Mean directional accuracy is 50.08%. These findings provide a statistically validated comparison of nine architectures, identify ModernTCN and PatchTST as leading performers under the evaluated conditions, and offer a reproducible benchmark for multi-asset financial time-series forecasting.
Abstract:Reliable comparison of deep learning architectures for financial forecasting requires controlled evaluation that separates architectural effects from hyperparameter tuning, random initialization, forecast horizon, and asset characteristics. We present a five-stage benchmark using equal-budget Bayesian hyperparameter optimization, category-level configuration freezing, three-seed final training, held-out metric aggregation, and rank-based statistical validation. Nine architectures from Transformer, MLP/linear, convolutional, and recurrent families are evaluated with direct multi-step prediction from five OHLCV features across 12 assets (four cryptocurrency, four foreign exchange, four equity-index instruments), using chronological 70/15/15 splits of 30,000 hourly windows per asset at 4- and 24-hour horizons. The benchmark includes 918 runs (270 optimization trials, 648 final training runs). Across 24 asset-horizon points, ModernTCN has mean rank 1.333 and the lowest RMSE in 18 points (75.0%), followed by PatchTST (mean rank 2.000). Global ranking differences are strongly supported by an Iman-Davenport test (F(8,184)=101.36, p<1e-15), with 33 of 36 Holm-adjusted pairwise comparisons significant. Rankings are consistent across horizons for all 12 assets (rho>=0.683; Stouffer p=3.47e-10), while errors increase at the longer horizon. Architecture explains 99.90% of raw error variance and 68.33% after within-slot normalization excluding LSTM; seed variance contributes at most 0.04% (0.02% without LSTM). Mean directional accuracy is 50.08%. These findings provide a statistically validated comparison of nine architectures, identify ModernTCN and PatchTST as leading performers under the evaluated conditions, and offer a reproducible benchmark for multi-asset financial time-series forecasting.
Statistical Finance (q-fin.ST); Machine Learning (cs.LG); General Finance (q-fin.GN)
arXiv:2603.16886 [q-fin.ST]
(or arXiv:2603.16886v2 [q-fin.ST] for this version)
https://doi.org/10.48550/arXiv.2603.16886 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 325 words