analiza aiqbench
Fin-RATE introduces a critical new benchmark for evaluating Large Language Models on complex SEC filings, addressing gaps in current testing methods that fail to capture multi-document synthesis and longitudinal tracking. The study reveals significant performance drops when models move beyond single-doc reasoning, highlighting specific failure modes like comparison hallucinations and temporal mismatches in financial analysis tasks. This work is essential for researchers aiming to improve the reliability of AI tools used in high-stakes regulatory environments where precision across multiple entities and time periods is paramount.
Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings
arXiv:2602.07294v4 Announce Type: replace-cross Abstract: With the increasing deployment of Large Language Models (LLMs) in the finance domain, LLMs are increasingly expected to parse complex regulatory disclosures. However, existing benchmarks often focus on isolated details, failing to reflect the complexity of professional analysis that requires synthesizing information across multiple documents, reporting periods, and corporate entities. Furthermore, these benchmarks do not disentangle whether errors arise from retrieval failures, generation inaccuracies, domain-specific reasoning mistakes, or misinterpretation of the query or context, making it difficult to precisely diagnose performance bottlenecks. To bridge these gaps, we introduce Fin-RATE, a benchmark built on U.S. Securities and Exchange Commission (SEC) filings and mirroring financial analyst workflows through three pathways: detail-oriented reasoning within individual disclosures, cross-entity comparison under shared topics, and longitudinal tracking of the same firm across reporting periods. We benchmark 17 leading LLMs, spanning open-source, closed-source, and finance-specialized models, under both ground-truth context and retrieval-augmented settings. Results show substantial performance degradation, with accuracy dropping by 18.60% and 14.35% as tasks shift from single-document reasoning to longitudinal and cross-entity analysis. This degradation is associated with increased comparison hallucinations, temporal and entity mismatches, and is further reflected in declines in reasoning quality and factual consistency--limitations that existing benchmarks have yet to formally categorize or quantify.
Abstract:With the increasing deployment of Large Language Models (LLMs) in the finance domain, LLMs are increasingly expected to parse complex regulatory disclosures. However, existing benchmarks often focus on isolated details, failing to reflect the complexity of professional analysis that requires synthesizing information across multiple documents, reporting periods, and corporate entities. Furthermore, these benchmarks do not disentangle whether errors arise from retrieval failures, generation inaccuracies, domain-specific reasoning mistakes, or misinterpretation of the query or context, making it difficult to precisely diagnose performance bottlenecks. To bridge these gaps, we introduce Fin-RATE, a benchmark built on U.S. Securities and Exchange Commission (SEC) filings and mirroring financial analyst workflows through three pathways: detail-oriented reasoning within individual disclosures, cross-entity comparison under shared topics, and longitudinal tracking of the same firm across reporting periods. We benchmark 17 leading LLMs, spanning open-source, closed-source, and finance-specialized models, under both ground-truth context and retrieval-augmented settings. Results show substantial performance degradation, with accuracy dropping by 18.60% and 14.35% as tasks shift from single-document reasoning to longitudinal and cross-entity analysis. This degradation is associated with increased comparison hallucinations, temporal and entity mismatches, and is further reflected in declines in reasoning quality and factual consistency--limitations that existing benchmarks have yet to formally categorize or quantify.
Computational Engineering, Finance, and Science (cs.CE); Artificial Intelligence (cs.AI)
arXiv:2602.07294 [cs.CE]
(or arXiv:2602.07294v4 [cs.CE] for this version)
https://doi.org/10.1145/3770855.3817528 DOI(s) linking to related resources
Text integral sub licența deschisă a publicației · 226 cuvinte
Subiecte conexe
SalQ-VLM: Fine-Grained Saliency-Guided Quantization for Vision-Language Models
arXiv · 22 sept. 2026
A Framework for Evaluating and Benchmarking Concept Drift Detection Methods
arXiv · 9 iun. 2026
Beyond Watching and Searching: Benchmarking Agents for Video-centered Deep Research Reports
arXiv · 24 oct. 2026
TRW: TRACE-RealWorld---An Auditable Consistency Contract for World Models as Materialized Views
arXiv · 28 iul. 2026