analiza aiqbench
This research demonstrates that while large language models excel at solving well-formed tax cases, they often fail to detect internal contradictions in defective inputs without explicit verification prompts. By integrating a simple self-check mechanism where the model verifies its own input before answering, systems can recover most missed errors with minimal accuracy loss and no additional training. This approach offers a practical pathway to building more reliable AI-driven compliance tools that can safely handle real-world legal data imperfections.
Solving versus Verifying: Catching Contradictions in Tax Reasoning Systems
arXiv:2609.05928v1 Announce Type: new Abstract: Large language models now compute correct tax liabilities on over 90% of well-formed cases in statutory benchmarks, which makes them candidates for the tax-advisory and compliance systems that consume such an answer directly. Real legal inputs, however, are frequently defective: required facts are missing, or stated facts contradict one another. Accuracy on clean benchmarks says nothing about how a model behaves then, and a system that computes straight through a defective input returns a confident number with no sign that anything is wrong. This raises two questions: does a model asked to solve a case abstain when the input is defective, and when it does not, can the same model catch the defect when asked instead to verify the input? We study six recent models on SARA-derived tax cases under missing-fact and contradictory-fact perturbations. The strongest models abstain when a fact is missing but compute through injected contradictions, returning the clean-input answer 63-76% of the time with no signal of the conflict; asked instead to verify the same input, they flag most of those contradictions. We wire that verification call into a simple contradiction gate: one extra call that abstains when the model reports a conflict. Across all six models it recovers most of the missed contradiction abstention at a clean-accuracy cost of at most about 5 percentage points, with no training and no external tooling. High accuracy on well-formed inputs is therefore an incomplete measure of reliability, and the detection the solver misses is cheaply recoverable with a single self-check.
Abstract:Large language models now compute correct tax liabilities on over 90% of well-formed cases in statutory benchmarks, which makes them candidates for the tax-advisory and compliance systems that consume such an answer directly. Real legal inputs, however, are frequently defective: required facts are missing, or stated facts contradict one another. Accuracy on clean benchmarks says nothing about how a model behaves then, and a system that computes straight through a defective input returns a confident number with no sign that anything is wrong. This raises two questions: does a model asked to solve a case abstain when the input is defective, and when it does not, can the same model catch the defect when asked instead to verify the input? We study six recent models on SARA-derived tax cases under missing-fact and contradictory-fact perturbations. The strongest models abstain when a fact is missing but compute through injected contradictions, returning the clean-input answer 63-76% of the time with no signal of the conflict; asked instead to verify the same input, they flag most of those contradictions. We wire that verification call into a simple contradiction gate: one extra call that abstains when the model reports a conflict. Across all six models it recovers most of the missed contradiction abstention at a clean-accuracy cost of at most about 5 percentage points, with no training and no external tooling. High accuracy on well-formed inputs is therefore an incomplete measure of reliability, and the detection the solver misses is cheaply recoverable with a single self-check.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2609.05928 [cs.CL]
(or arXiv:2609.05928v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2609.05928 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 349 cuvinte
Subiecte conexe
Automated grounding verification of AI statements about chest radiographs
arXiv · 4 nov. 2026
VERDI:Verified Contradictions for Continual World-Model Optimization
arXiv · 20 oct. 2026
Mandela-Bench: Multimodal Models Remember Canonical Images Instead of Seeing Them
arXiv · 29 sept. 2026
Do Large Language Models Hallucinate Electric Fata Morganas?
arXiv · 21 aug. 2026