Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded-Reasoning

Today, we are releasing OfficeQA Pro V2, a new benchmark designed to evaluate whether AI agents can generalize to unfamiliar, enterprise-style grounded-reasoning tasks.

Seven months ago, we introduced the OfficeQA benchmark to measure how well AI systems answer analytical questions using evidence from large document collections, an extremely common and important enterprise task that we found agents struggled with. Since its introduction, OfficeQA, and its frontier subset, OfficeQA Pro, has become an important measure for frontier model and agent capabilities, driving progress in document retrieval, parsing, and analytical reasoning. But this progress raises a fundamental question: do these improvements reflect broader advances in grounded reasoning, or progress specific to one corpus and task distribution? This distinction matters in enterprise settings, where agents rarely operate on a single, stable document collection.

Our new benchmark, OfficeQA Pro V2, is designed to test that generalization directly. We initially developed OfficeQA Pro V2 as the benchmark for the inaugural Databricks Grounded Reasoning Cup, a live competition in which 11 academic teams, supported by OpenAI, Anthropic, and Google DeepMind, built agents that were evaluated on a previously unseen corpus and set of grounded-reasoning tasks. The benchmark contains 90 questions grounded in approximately 120,000 pages from the U.S. Treasury's Accounts of Receipts and Expenditures, which was released as a cohesive dataset by the U.S. Treasury for the first time in conjunction with the 250th anniversary of the United States. Now, we are excited to release the benchmark to AI practitioners more broadly to use for their own development.

Accuracy on OfficeQA Pro V2 using model-provider harnesses (Claude Code for Anthropic models and Codex for OpenAI models) versus the Databricks Genie. Across the four matched models, Genie improved mean accuracy by 15.3 percentage points, from 37.5% to 52.8%.

OfficeQA Pro V2 remains challenging for today's AI systems. Out-of-the-box frontier agents using Claude Code with Claude Opus 4.8 and Claude Fable 5, or Codex with GPT-5.5 and GPT-5.6 Sol, achieved an average accuracy of 37.5%. Agents developed specifically for the Grounded Reasoning Cup performed better, averaging 41.1%, with the winning team reaching 63.3%. We also evaluated Databricks Genie, Databricks' AI agent designed to answer complex questions over enterprise data, using the same underlying models. Genie, which also uses Databricks' ai_parse to pre-parse the document corpus, improved accuracy by an average of 24.0 percentage points over the default harnesses, with the strongest configuration reaching 60%. Together, these results show that grounded reasoning remains far from solved, but that the right agent harness can unlock significant gains from existing frontier models.

Below, we describe in more detail how agents perform on OfficeQA Pro V2, how it was built, and how it differs from the original OfficeQA benchmark.

Cost versus correctness for model + harness combinations on OfficeQA Pro V2. Models using Databricks Genie dominate the Pareto frontier, achieving higher quality at a lower cost. Per-rollout costs do not include the one-time, upfront cost of parsing the corpus with ai_parse.

We first evaluated frontier models using each model provider's associated harness, with Codex for GPT models and Claude Code for Claude models, using the same deterministic exact-match metric with 0.0% tolerance metric as used for OfficeQA. Across the five models evaluated under both harnesses, these baseline configurations achieved 26.0% accuracy on average. Performance varied substantially by model, and higher cost did not consistently translate into greater accuracy. For example, Sonnet 5 on Claude Code scored 15.6% at $5.01 per rollout, while GPT-5.6 Sol on Codex reached 33.3% at a comparable $4.70.

Using these models within Genie produced substantial gains. Across matched model comparisons, Genie improved accuracy by 24.0 percentage points on average (a 92% relative improvement). Genie configurations using GPT-5.6 Luna, GPT-5.6 Terra, and Claude Fable 5 also dominate the cost-quality Pareto frontier, demonstrating that these improvements do not require trading efficiency for accuracy. In the case of Claude Fable 5, switching to Genie improves performance by 14.4 percentage points (+32% relative) while reducing cost by ~9x. From inspecting traces, we find that Fable 5 is prone to looping attempts to parse many documents, which leads to long rollouts costing $37.36 on average. Genie parses documents using ai_parse document, which enables the model to efficiently identify the correct information from each page, thereby saving cost and improving performance.

While these harness improvements lead to substantial gains, significant headroom remains in OfficeQA Pro V2. Systems continue to exhibit similar failure modes to those we observed in the original OfficeQA – parsing fidelity, incorrect temporal reconciliation as accounting conventions change over time, and misinterpretation of entity scope or category granularity.

To truly test generalization, we needed a new corpus that still reflected the same enterprise-relevant skills OfficeQA was designed to measure: parsing complex documents, retrieving the right evidence, and performing analytical reasoning grounded in that information.

We worked with the U.S. Treasury to build the benchmark around a new corpus: the U.S. Federal Accounts of Receipts and Expenditures, which the Treasury was preparing to release as a unified dataset for the first time in conjunction with the 250th anniversary of the United States. The corpus consists of roughly 1,400 PDFs and 120,000 pages containing detailed U.S. accounting records, spanning 1793 through 2024.

Like the U.S. Treasury Bulletins used in the original OfficeQA benchmark, this new corpus reflects many of the challenges common across enterprise document collections: dense tables and charts, values revised over time, evolving reporting conventions, and deep institutional knowledge. These complexities also translate into a challenging evaluation: as with the original OfficeQA benchmark, we find that baseline frontier agents struggle to achieve consistently high accuracy on OfficeQA Pro V2.

Synthetic Data Pipeline used to build OfficeQA Pro V2.

When we created the original OfficeQA benchmark in late 2025, the process was highly manual. Human annotators constructed questions and answers by hand while combing through a corpus comprising 89,000 U.S. Treasury Bulletin pages, often requiring numerous rounds of human review in order to ensure question quality. Although this approach eventually produced a high-quality benchmark, it was time-consuming, costly, and difficult to scale. Since then, we have developed techniques to automate the creation of rigorous benchmarks much more efficiently and reliably.

To build OfficeQA Pro V2, we leveraged asynth, our internal library for building synthetic data-generation pipelines. This enabled us to quickly and scalably create a diverse, verifiable benchmark that accurately reflected the challenges of grounded reasoning using a systematic process for generating and vetting questions:

Inputs. To promote sample diversity, the synthesizer was seeded with a randomly selected time period, analytical method, target number of source documents, and salient topic drawn from a list developed with our partner, USAFacts, to reflect questions that analysts might naturally ask of the corpus.

Synthesis. For each candidate question, a synthesis agent searched the corpus for evidence matching these criteria and generated a sample supported by a traceable chain of grounded reasoning.

Verification. Next, samples went through several quality gates to audit for source fidelity, analytical relevance, diversity, and answerability. Independent solver agents attempted the question from scratch, produc