aiqbench analysis
This research advances Retrieval-Augmented Generation (RAG) by introducing Parametric Knowledge Precision (PKP), a new metric designed to accurately evaluate factuality in specialized technical domains like electronics engineering. The study demonstrates that fine-tuning suppresses hallucinations without necessarily reinforcing parametric knowledge, challenging standard evaluation metrics that often mislead on long-form generation tasks. These findings are critical for developers seeking reliable AI systems capable of generating high-fidelity requirements documents beyond generic question-answering scenarios.
Parametric Knowledge in RAG-SFT for Domain-Specific Document Generation
arXiv:2603.23047v2 Announce Type: replace Abstract: Retrieval-Augmented Generation (RAG) fine-tuning has shown substantial improvements over vanilla RAG, yet most studies target document question answering, leaving open whether these gains transfer to specialized tasks. We study supervised RAG fine-tuning (RAG-SFT) for requirements document generation in the electronics engineering domain, adapting two 7B models under two different training data strategies. Because Rouge and BertScore poorly capture factuality on long technical text, we introduce C-FEX, a claim-based evaluation pipeline that attributes each response claim to its origin (augmented prompt or reference response), and propose Parametric Knowledge Precision (PKP), which isolates claims originating from the model's weights and measures their correctness. We show that a prior metric to assess parametric knowledge decomposes as PKP $\times$ PR, separating the rate of parametric output (PR) from its quality (PKP). Empirically, fine-tuned 7B models match or exceed a 72B baseline; standard metrics disagree with claim-based factuality and can mislead about fine-tuning gains; and, fine-tuning does not reinforce correct parametric knowledge but suppresses hallucination---models speak from their weights less often but far more reliably.
Abstract:Retrieval-Augmented Generation (RAG) fine-tuning has shown substantial improvements over vanilla RAG, yet most studies target document question answering, leaving open whether these gains transfer to specialized tasks. We study supervised RAG fine-tuning (RAG-SFT) for requirements document generation in the electronics engineering domain, adapting two 7B models under two different training data strategies. Because Rouge and BertScore poorly capture factuality on long technical text, we introduce C-FEX, a claim-based evaluation pipeline that attributes each response claim to its origin (augmented prompt or reference response), and propose Parametric Knowledge Precision (PKP), which isolates claims originating from the model's weights and measures their correctness. We show that a prior metric to assess parametric knowledge decomposes as PKP $\times$ PR, separating the rate of parametric output (PR) from its quality (PKP). Empirically, fine-tuned 7B models match or exceed a 72B baseline; standard metrics disagree with claim-based factuality and can mislead about fine-tuning gains; and, fine-tuning does not reinforce correct parametric knowledge but suppresses hallucination---models speak from their weights less often but far more reliably.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computational Engineering, Finance, and Science (cs.CE)
arXiv:2603.23047 [cs.CL]
(or arXiv:2603.23047v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2603.23047 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 272 words
Related coverage
Knowledge Graph Integration With Poro2 For Enriching Medical Text Processing
AMD · Sep 15, 2026
Retrievable Gradients: Continual Post-Training Without Cumulative Weight Drift
arXiv · Oct 15, 2026
Retrievable Gradients: Continual Post-Training Without Cumulative Weight Drift
arXiv · Jun 17, 2026
Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval
arXiv · Oct 6, 2026