aiqbench analysis
This study evaluates large language models on the QIAS 2026 shared task for Arabic Islamic inheritance reasoning, highlighting a significant reliability gap between commercial and open-source architectures. Commercial models, specifically Gemini 2.5 Flash, demonstrate superior consistency in multi-step legal interpretation and numerical computation compared to unstable open-source alternatives. The findings underscore current limitations in deploying open-weight models for high-stakes domains requiring strict adherence to complex rule-based logic without extensive fine-tuning.
Which Models Perform Better in Inheritance Reasoning?
arXiv:2606.13751v4 Announce Type: replace Abstract: This paper presents the participation of team PSL in the QIAS 2026 Shared Task on Arabic Islamic inheritance reasoning. The task evaluates the ability of large language models to solve inheritance cases that require legal interpretation, multi-step reasoning, and precise numerical computation. We compare \textit{commercial} and \textit{open-source} models under a unified prompting strategy to assess their effectiveness in structured legal reasoning with minimal task-specific adaptation. \\ Our results show a clear gap in reliability between the two model families. Commercial models demonstrate stronger performance in identifying eligible heirs, applying exclusion rules, and maintaining consistency across reasoning steps. In contrast, open-source models exhibit greater instability, particularly in cases involving dependent legal decisions and fractional share adjustments. The best performance is achieved by \textit{Gemini 2.5 Flash}, with an MRE of $0.989$.
Abstract:This paper presents the participation of team PSL in the QIAS 2026 Shared Task on Arabic Islamic inheritance reasoning. The task evaluates the ability of large language models to solve inheritance cases that require legal interpretation, multi-step reasoning, and precise numerical computation. We compare \textit{commercial} and \textit{open-source} models under a unified prompting strategy to assess their effectiveness in structured legal reasoning with minimal task-specific adaptation. \\ Our results show a clear gap in reliability between the two model families. Commercial models demonstrate stronger performance in identifying eligible heirs, applying exclusion rules, and maintaining consistency across reasoning steps. In contrast, open-source models exhibit greater instability, particularly in cases involving dependent legal decisions and fractional share adjustments. The best performance is achieved by \textit{Gemini 2.5 Flash}, with an MRE of $0.989$.
Computation and Language (cs.CL)
arXiv:2606.13751 [cs.CL]
(or arXiv:2606.13751v4 [cs.CL] for this version)
Full text under the publisher's open licence · 141 words
Related coverage
[AINews] not much happened today
Latent Space · Sep 18, 2026
[AINews] Claude Opus 5.5, the new default model for AINews — and everybody cuts prices 40-50%
Latent Space · Sep 23, 2026
Construct validity of Claude Opus 4.8's System Card – A commentary
LessWrong · Jun 12, 2026
Hyperloom: A Multi-Agent Harness for Autonomous Inference Optimization on AMD GPUs
AMD · Sep 21, 2026