analiza aiqbench
Deontic Agentic Reasoning (DAR) demonstrates that allowing models to interact with statutes on demand can improve performance on complex rule-based tasks like tax liability or immigration appeals. However, this approach introduces non-uniform trade-offs, where weaker models often suffer degraded numerical accuracy and significantly higher token consumption. For practitioners, the key takeaway is that agentic harnesses offer a viable path forward for specific reasoning domains but require careful evaluation of cost and capability balance before deployment.
DAR: Deontic Reasoning with Agentic Harnesses
arXiv:2606.05009v1 Announce Type: new Abstract: Deontic reasoning is the task of answering questions by applying explicit rules and policies to case-specific facts, for example computing tax liability under a statute or determining the outcome of an immigration appeal. A key technical challenge for LLM-based deontic reasoning is that the relevant ruleset can be long and cross-referenced, so models may still fail to locate the rules needed for a particular reasoning step. We introduce Deontic Agentic Reasoning (DAR), an agentic reasoning setup in which the model interacts with the statutes on demand. We evaluate DAR under multiple harnesses on hard subsets of DeonticBench. Across these settings, we find that agentic harnesses can push the frontier on deontic reasoning tasks, but improvements are not uniform: weaker models often degrade on numerical tasks while consuming far more tokens.
Abstract:Deontic reasoning is the task of answering questions by applying explicit rules and policies to case-specific facts, for example computing tax liability under a statute or determining the outcome of an immigration appeal. A key technical challenge for LLM-based deontic reasoning is that the relevant ruleset can be long and cross-referenced, so models may still fail to locate the rules needed for a particular reasoning step. We introduce Deontic Agentic Reasoning (DAR), an agentic reasoning setup in which the model interacts with the statutes on demand. We evaluate DAR under multiple harnesses on hard subsets of DeonticBench. Across these settings, we find that agentic harnesses can push the frontier on deontic reasoning tasks, but improvements are not uniform: weaker models often degrade on numerical tasks while consuming far more tokens.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2606.05009 [cs.CL]
(or arXiv:2606.05009v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2606.05009 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 228 cuvinte
Subiecte conexe
HarnessMM: Recursive Self-Improvement for Harness Token Efficiency
arXiv · 27 oct. 2026
Prompt-Induced Waste in Large Reasoning Models: A Preregistered Two-Harness Benchmark of Coding Agents
arXiv · 4 aug. 2026
Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents
arXiv · 27 oct. 2026
[AINews] GPT-6 Astra: OpenAI’s biggest LLM launch of all time
Latent Space · 4 sept. 2026