analiza aiqbench
This paper introduces QuantHarness, a framework designed to ensure the reliability and fidelity of financial strategies generated by LLM agents throughout their development trajectory. By enforcing strict boundaries on information access and actions at each stage, it addresses critical risks like data leakage or rule misinterpretation that standard metrics often miss. The authors also present FinRelBench, a new benchmark specifically tailored to evaluate these reliability attributes beyond simple task success rates.
Reliable Before Profitable: QuantHarness for Governing Financial Strategy Development with LLM Agents
arXiv:2610.26101v1 Announce Type: new Abstract: LLM agents can turn natural-language financial research requests into executable and backtested strategies, but a successful run may still implement the wrong rule, use information unavailable at decision time, or obscure how repairs and candidate selection changed the strategy. Because these failures arise throughout development, final code and performance metrics alone are insufficient to establish either fidelity to the request or the validity of the development process. We therefore address the reliability of financial strategy development at the trajectory level: the system must control how intermediate outputs become downstream state and preserve the evidence needed to justify the terminal outcome. We introduce QuantHarness, a two-layer agent harness in which a finance-specific Workflow structures specification, data grounding, construction, testing, and review, while Harness Core enforces stage-specific information, action, and commitment boundaries. We also introduce FinRelBench, a benchmark that evaluates strategy-development reliability through execution success, requirement satisfaction, and runtime evidence. QuantHarness achieves the highest Task Success at 90.72\%, compared with 74.23\%--84.54\% for the comparison systems. Controlled stress tests show that its boundaries contain targeted violations, while structured run records improve evidence retrieval and support decision replay under the evaluated protocol.
Abstract:LLM agents can turn natural-language financial research requests into executable and backtested strategies, but a successful run may still implement the wrong rule, use information unavailable at decision time, or obscure how repairs and candidate selection changed the strategy. Because these failures arise throughout development, final code and performance metrics alone are insufficient to establish either fidelity to the request or the validity of the development process. We therefore address the reliability of financial strategy development at the trajectory level: the system must control how intermediate outputs become downstream state and preserve the evidence needed to justify the terminal outcome. We introduce QuantHarness, a two-layer agent harness in which a finance-specific Workflow structures specification, data grounding, construction, testing, and review, while Harness Core enforces stage-specific information, action, and commitment boundaries. We also introduce FinRelBench, a benchmark that evaluates strategy-development reliability through execution success, requirement satisfaction, and runtime evidence. QuantHarness achieves the highest Task Success at 90.72\%, compared with 74.23\%--84.54\% for the comparison systems. Controlled stress tests show that its boundaries contain targeted violations, while structured run records improve evidence retrieval and support decision replay under the evaluated protocol.
Artificial Intelligence (cs.AI)
arXiv:2610.26101 [cs.AI]
(or arXiv:2610.26101v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.26101 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
finrelbench_inventory.csv
finresearchsuite_inventory.csv
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 284 cuvinte