aiqbench analysis
This study experimentally evaluates how fine-tuning and retrieval-augmented generation (RAG) can enhance large language models for legal reasoning in Austrian VAT law, demonstrating their utility for automating routine analysis while highlighting persistent risks of hallucination. The research concludes that current prototypes require human oversight due to limitations in handling implicit client knowledge, positioning LLMs as supportive tools rather than fully autonomous decision-makers in sensitive legal domains.
Using Large Language Models for Legal Decision-Making in Austrian Value-Added Tax Law: A Comparative Study
arXiv:2507.08468v2 Announce Type: replace Abstract: This paper provides an experimental evaluation of the capability of large language models (LLMs) to assist in legal decision-making within the framework of Austrian and European Union value-added tax (VAT) law. In tax consulting practice, clients often describe cases in natural language, making LLMs a prime candidate for supporting automated decision-making and reducing the workload of tax professionals. Given the requirement for legally grounded and well-justified analyses, the propensity of LLMs to hallucinate presents a considerable challenge. The experiments focus on two common methods for enhancing LLM performance: fine-tuning and retrieval-augmented generation (RAG). In this study, these methods are applied on both textbook cases and real-world cases from a tax consulting firm to systematically determine the best configurations of LLM-based systems and assess the legal-reasoning capabilities of LLMs. The findings highlight the potential of using LLMs to support tax consultants by automating routine tasks and providing initial analyses, although current prototypes are not ready for full automation due to the sensitivity of the legal domain. The findings indicate that LLMs, when properly configured, can effectively support tax professionals in VAT tasks and provide legally grounded justifications for decisions. However, limitations remain regarding the handling of implicit client knowledge and context-specific documentation, underscoring the need for future integration of structured background information.
Abstract:This paper provides an experimental evaluation of the capability of large language models (LLMs) to assist in legal decision-making within the framework of Austrian and European Union value-added tax (VAT) law. In tax consulting practice, clients often describe cases in natural language, making LLMs a prime candidate for supporting automated decision-making and reducing the workload of tax professionals. Given the requirement for legally grounded and well-justified analyses, the propensity of LLMs to hallucinate presents a considerable challenge. The experiments focus on two common methods for enhancing LLM performance: fine-tuning and retrieval-augmented generation (RAG). In this study, these methods are applied on both textbook cases and real-world cases from a tax consulting firm to systematically determine the best configurations of LLM-based systems and assess the legal-reasoning capabilities of LLMs. The findings highlight the potential of using LLMs to support tax consultants by automating routine tasks and providing initial analyses, although current prototypes are not ready for full automation due to the sensitivity of the legal domain. The findings indicate that LLMs, when properly configured, can effectively support tax professionals in VAT tasks and provide legally grounded justifications for decisions. However, limitations remain regarding the handling of implicit client knowledge and context-specific documentation, underscoring the need for future integration of structured background information.
43 pages, 5 figures, 6 tables. Published open access in Journal of Business Analytics under a CC BY-NC-ND 4.0 license. Author's accepted manuscript (incorporating peer-review changes)
Computation and Language (cs.CL)
arXiv:2507.08468 [cs.CL]
(or arXiv:2507.08468v2 [cs.CL] for this version)
https://doi.org/10.1080/2573234X.2026.2720626 DOI(s) linking to related resources
Full text under the publisher's open licence · 255 words
Related coverage
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · Sep 18, 2026
The Continual Constitutional Convention: How language models revise the documents that govern them
AI Alignment Forum · Oct 13, 2026
Why I Left Google DeepMind
LessWrong · Jul 15, 2026
Deconfusing Corrigibility
AI Alignment Forum · Oct 17, 2026