aiqbench analysis
This study critically examines the reproducibility crisis in LLM-based trading research, revealing that evaluation assumptions regarding execution timing and transaction costs often obscure economic interpretability more than agent architecture flaws do. The authors argue that advancing the field requires establishing rigorous reporting standards for friction modeling and temporal splits rather than solely focusing on better agent design. By auditing 30 primary studies, the work highlights how explicit modeling of market frictions is essential for generating comparable and economically valid results in algorithmic finance.
Beyond Agent Architecture: Execution Assumptions and Reproducibility in LLM-Based Trading Systems
arXiv:2606.08285v2 Announce Type: replace Abstract: Large language models (LLMs) and agentic systems are increasingly proposed for financial trading, yet their reported performance remains difficult to compare because studies vary in data provenance, temporal split discipline, execution timing, turnover treatment, and transaction-cost modeling. This article presents a targeted topical review and reproducibility audit of execution realism in LLM-based trading research. A coded evidence matrix covering 30 trade-relevant primary studies is used to assess point-in-time controls, split transparency, held-out evaluation, cost and turnover treatment, execution semantics, universe definition, and artifact release. Across the audited sample, architecture reporting is generally clearer than the evaluation assumptions needed to judge whether a trading result is economically interpretable or reproducible. A 10-equity worked example is included only as a methodological scaffold to illustrate how explicit friction and timing choices can materially compress active-strategy results. The main conclusion is that the next useful step for LLM trading research is not only better agent design, but also clearer reporting standards for execution realism, reproducibility, and evaluation comparability.
Abstract:Large language models (LLMs) and agentic systems are increasingly proposed for financial trading, yet their reported performance remains difficult to compare because studies vary in data provenance, temporal split discipline, execution timing, turnover treatment, and transaction-cost modeling. This article presents a targeted topical review and reproducibility audit of execution realism in LLM-based trading research. A coded evidence matrix covering 30 trade-relevant primary studies is used to assess point-in-time controls, split transparency, held-out evaluation, cost and turnover treatment, execution semantics, universe definition, and artifact release. Across the audited sample, architecture reporting is generally clearer than the evaluation assumptions needed to judge whether a trading result is economically interpretable or reproducible. A 10-equity worked example is included only as a methodological scaffold to illustrate how explicit friction and timing choices can materially compress active-strategy results. The main conclusion is that the next useful step for LLM trading research is not only better agent design, but also clearer reporting standards for execution realism, reproducibility, and evaluation comparability.
Artificial Intelligence (cs.AI); Computational Engineering, Finance, and Science (cs.CE); Computational Finance (q-fin.CP); Trading and Market Microstructure (q-fin.TR)
arXiv:2606.08285 [cs.AI]
(or arXiv:2606.08285v2 [cs.AI] for this version)
Full text under the publisher's open licence · 189 words
Related coverage
Beyond Agent Architecture: Execution Assumptions and Reproducibility in LLM-Based Trading Systems
arXiv · Jun 9, 2026
AI and Quantum Computing Briefing
livetradingnews.com · Aug 4, 2026
Benchmarking Kimi-K3 Across vLLM, SGLang, and ATOM on MI350X
AMD · Sep 22, 2026
DI Series: Serving Kimi-K3 MXFP4 Wide-EP Disaggregated on AMD Instinct MI300X and MI325X
AMD · Oct 8, 2026