analiza aiqbench
This research addresses the critical challenge of mitigating factual errors in Large Language Models when answering specific domain queries, such as Islamic jurisprudence. By fine-tuning an open-source model to detect hallucinations and select verified answers from candidate options, the study demonstrates a viable path toward more reliable AI applications in sensitive knowledge domains. The results highlight that while detecting false statements is highly accurate, distinguishing between plausible alternatives remains a significant hurdle for current systems.
Detecting Hallucinations and Recovering Verified Answers in Arabic Islamic Question Answering
arXiv:2608.03720v1 Announce Type: new Abstract: Large language models can generate fluent responses to Islamic questions while introducing factual errors that are difficult to identify. This paper presents our system for \textsc{HalluScoring 2026} Task 2.1, \textit{Islamic Hallucination Detection and Find the Truth}. The task requires a unified two-step prediction: determining whether an Arabic answer generated by an LLM is hallucinated and selecting the verified answer from six closely related candidate options. We use the Islamic knowledge dataset provided by the shared task, which contains 600 question--answer instances, including 341 hallucinated and 259 non-hallucinated answers. Our system is based on the fine-tuned \texttt{google/gemma-4-12B-it} model and uses deterministic decoding during inference. The generated outputs are normalized to extract the hallucination label and the selected option. The system achieves a Macro-F1 score of 0.928 and a label accuracy of 0.935 for hallucination detection, together with an option accuracy of 0.895 for answer selection. These results yield a combined score of 0.912, demonstrating strong performance across both stages of the task. The lower option-selection accuracy indicates that distinguishing the verified answer from plausible alternatives remains more challenging than detecting hallucinated responses.
Abstract:Large language models can generate fluent responses to Islamic questions while introducing factual errors that are difficult to identify. This paper presents our system for \textsc{HalluScoring 2026} Task 2.1, \textit{Islamic Hallucination Detection and Find the Truth}. The task requires a unified two-step prediction: determining whether an Arabic answer generated by an LLM is hallucinated and selecting the verified answer from six closely related candidate options. We use the Islamic knowledge dataset provided by the shared task, which contains 600 question--answer instances, including 341 hallucinated and 259 non-hallucinated answers. Our system is based on the fine-tuned \texttt{google/gemma-4-12B-it} model and uses deterministic decoding during inference. The generated outputs are normalized to extract the hallucination label and the selected option. The system achieves a Macro-F1 score of 0.928 and a label accuracy of 0.935 for hallucination detection, together with an option accuracy of 0.895 for answer selection. These results yield a combined score of 0.912, demonstrating strong performance across both stages of the task. The lower option-selection accuracy indicates that distinguishing the verified answer from plausible alternatives remains more challenging than detecting hallucinated responses.
Computation and Language (cs.CL)
arXiv:2608.03720 [cs.CL]
(or arXiv:2608.03720v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.03720 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 276 cuvinte
Subiecte conexe
HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering
arXiv · 23 iul. 2026
HalluTruthQA-4K: A Fine-Grained Corpus and Annotation Process for Arabic Hallucination Detection and Truth Verification
arXiv · 6 aug. 2026
Halluscoring 2026: The first shared task on llms hallucination detection and answer verification
arXiv · 2 oct. 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · 19 mai 2026