aiqbench analysis
This paper investigates the efficacy of confidence-based auditing strategies for LLM-generated knowledge graphs, revealing that superior error prediction does not always translate into significant audit improvements due to constraints like repairability and label validity. By modeling non-adaptive auditing as a budgeted covering problem, the authors characterize minimal decision-relevant error statistics and demonstrate scenarios where joint planning models outperform independent block approaches in reducing residual errors. The study provides critical insights for practitioners aiming to optimize human-in-the-loop review processes without overestimating the marginal gains of better confidence estimates.
When Does Better Confidence Improve a Knowledge-Graph Audit?
arXiv:2611.05877v1 Announce Type: new Abstract: Confidence estimates guide human review in LLM-built knowledge graphs, but better error prediction can yield little improvement in an audit. We study this gap through repairability, selection headroom and label validity. Modelling non-adaptive auditing as budgeted covering, we characterise the minimal decision-relevant error statistics and bound the regret of scalar-preserving first-order planning by a structural mass $\Delta^{\mathrm{cl}}$. We simulate ideal repair against benchmark-aligned labels for five extraction configurations and two cross-document graphs on Re-DocRED, followed by frozen confirmation on test and replication on 500 hash-selected training documents. On the held-out cross-document graphs, an error-rate $\times$ fan-out queue leaves 3.98-4.74 percentage points more errors than a joint model with the same actions. By comparison, opening shared actions for that joint planner reduces residual errors by 0.89-1.43 points relative to its own fact-only plans, in point estimates. Equivalence between joint and independent-block models remains inconclusive. Generic LLM verification improves three of four model-budget configurations on each held-out set. LLM adjudication identifies possible benchmark omissions.
Title:When Does Better Confidence Improve a Knowledge-Graph Audit?
Abstract:Confidence estimates guide human review in LLM-built knowledge graphs, but better error prediction can yield little improvement in an audit. We study this gap through repairability, selection headroom and label validity. Modelling non-adaptive auditing as budgeted covering, we characterise the minimal decision-relevant error statistics and bound the regret of scalar-preserving first-order planning by a structural mass $\Delta^{\mathrm{cl}}$. We simulate ideal repair against benchmark-aligned labels for five extraction configurations and two cross-document graphs on Re-DocRED, followed by frozen confirmation on test and replication on 500 hash-selected training documents. On the held-out cross-document graphs, an error-rate $\times$ fan-out queue leaves 3.98-4.74 percentage points more errors than a joint model with the same actions. By comparison, opening shared actions for that joint planner reduces residual errors by 0.89-1.43 points relative to its own fact-only plans, in point estimates. Equivalence between joint and independent-block models remains inconclusive. Generic LLM verification improves three of four model-budget configurations on each held-out set. LLM adjudication identifies possible benchmark omissions.
27 pages, 4 figures, 19 tables. Ancillary files include a public rendering of the confirmation protocol and summary results
Computation and Language (cs.CL)
arXiv:2611.05877 [cs.CL]
(or arXiv:2611.05877v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2611.05877 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
confirmation_protocol.md
diagnostics.json
shared_action_diagnostics.json
test_confirm.json
train500_confirm.json
(2 additional files not shown) You must enabled JavaScript to view entire file list.
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 303 words