aiqbench analysis
This study evaluates CRISS, a retrieval-augmented generation (RAG) system designed to assist cancer registrars by grounding responses in national coding standards. The research demonstrates that RAG configurations significantly outperform non-RAG baselines in accuracy and semantic similarity, particularly as question complexity increases. By comparing open-weight, proprietary, and local models, the work highlights the trade-offs between performance and caution required for high-stakes medical data interpretation.
CRISS: A Retrieval-Augmented AI Chatbot for Assisting Cancer Registrars
arXiv:2609.29075v1 Announce Type: cross Abstract: Cancer registrars, including Oncology Data Specialists (ODSs), must interpret complex and frequently updated coding and staging standards. We developed CRISS (Cancer Registry Intelligent Support System), a retrieval-augmented generation (RAG) conversational assistant that provides rapid, citation-supported access to registry guidance. This study evaluated whether CRISS could (1) support accurate and citation-supported responses, (2) improve access to and interpretation of relevant guidance, and (3) support training/helpdesk use while preserving human oversight of final abstraction decisions. We built a domain-specific knowledge base from national cancer registry standards, segmented into metadata-tagged passages and indexed as dense embeddings. Retrieved passages were used to generate citation-grounded responses through a large language model (LLM). Open-weight, proprietary, and non-RAG baseline models across Gemini and GPT families were evaluated on easy, medium, and hard registry questions using an LLM-as-a-Judge protocols. RAG configurations consistently outperformed non-RAG approaches, especially as question difficulty increased. Mean grounding scores for RAG were 0.62/0.56/0.59 across easy/medium/hard tiers versus 0.29/0.26/0.29 for non-RAG. RAG models also achieved higher semantic-similarity scores overall. Proprietary RAG models performed strongest on easy and medium questions, while local RAG models ranked highest on hard questions and proprietary models were generally more cautious. Domain-specific RAG improved evidence grounding and response quality for cancer registry questions while enabling citation-supported assistance across complexity levels. CRISS demonstrates the potential of human-centered, citation-grounded AI to support cancer registrars while preserving human oversight for final coding decisions.
Abstract:Cancer registrars, including Oncology Data Specialists (ODSs), must interpret complex and frequently updated coding and staging standards. We developed CRISS (Cancer Registry Intelligent Support System), a retrieval-augmented generation (RAG) conversational assistant that provides rapid, citation-supported access to registry guidance. This study evaluated whether CRISS could (1) support accurate and citation-supported responses, (2) improve access to and interpretation of relevant guidance, and (3) support training/helpdesk use while preserving human oversight of final abstraction decisions. We built a domain-specific knowledge base from national cancer registry standards, segmented into metadata-tagged passages and indexed as dense embeddings. Retrieved passages were used to generate citation-grounded responses through a large language model (LLM). Open-weight, proprietary, and non-RAG baseline models across Gemini and GPT families were evaluated on easy, medium, and hard registry questions using an LLM-as-a-Judge protocols. RAG configurations consistently outperformed non-RAG approaches, especially as question difficulty increased. Mean grounding scores for RAG were 0.62/0.56/0.59 across easy/medium/hard tiers versus 0.29/0.26/0.29 for non-RAG. RAG models also achieved higher semantic-similarity scores overall. Proprietary RAG models performed strongest on easy and medium questions, while local RAG models ranked highest on hard questions and proprietary models were generally more cautious. Domain-specific RAG improved evidence grounding and response quality for cancer registry questions while enabling citation-supported assistance across complexity levels. CRISS demonstrates the potential of human-centered, citation-grounded AI to support cancer registrars while preserving human oversight for final coding decisions.
21 pages, 13 figures, 7 tables. Keywords: cancer registry, retrieval-augmented generation, large language models, conversational AI, clinical informatics, oncology data specialists, medical question answering, AI safety, clinical decision support
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv:2609.29075 [cs.AI]
(or arXiv:2609.29075v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2609.29075 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 280 words
Related coverage
What is an AI Copilot?
Databricks · Aug 28, 2026
Comparing the Best AI Chatbots for Work in 2026
Slack Blog · Sep 28, 2026
Beyond the Chatbot: Co-Learning and Co-Teaching through a Dual-Persona Generative-AI Assistant
arXiv · Aug 27, 2026
AI customer service: strategy, agents, and solutions guide
Databricks · Jul 27, 2026