analiza aiqbench
This research introduces IAR, a novel three-stage post-training framework designed to internalize document knowledge into large language models for retrieval-free question answering. By separating structured injection, QA alignment, and general capability recovery, the method successfully improves domain accuracy while mitigating the typical degradation of general performance seen in standard supervised fine-tuning. The study demonstrates significant gains across multiple model families, offering a practical solution for deploying LLMs with specialized knowledge bases without relying on external retrieval systems.
Inject, Align, Recover: Staged Post-Training for Retrieval-Free Document Knowledge Internalization
arXiv:2608.20281v1 Announce Type: new Abstract: Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time. We study this setting as document knowledge internalization: converting a fixed corpus into usable parametric knowledge for retrieval-free question answering. We propose IAR (Inject, Align, and Recover), a three-stage post-training framework that separates structured document knowledge injection, QA behavior alignment, and general ability recovery. Unlike conventional continued pretraining, Inject converts source documents into continuation, rewrite, and instruction-conditioned reconstruction objectives. Align then adapts the injected model with answer-only QA supervision, while Recover merges the domain-adapted model with the base instruction model to recover general capabilities. Across Common Corpus (CC) and CCI, and across Llama, Phi, Qwen, and SmolLM model families, IAR improves the domain-primary domain-general frontier for retrieval-free document internalization. In the main comparison, IAR improves over Vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance across IFEval, MMLU, and MSBench. Extended CC baselines show that LoRA and FAPM can win individual general metrics, but among methods that also reach leading or near-leading domain internalization, IAR retains one of the strongest general profiles.
Abstract:Large language models often fail to answer questions about a bounded document collection when the source documents are not retrieved at inference time. We study this setting as document knowledge internalization: converting a fixed corpus into usable parametric knowledge for retrieval-free question answering. We propose IAR (Inject, Align, and Recover), a three-stage post-training framework that separates structured document knowledge injection, QA behavior alignment, and general ability recovery. Unlike conventional continued pretraining, Inject converts source documents into continuation, rewrite, and instruction-conditioned reconstruction objectives. Align then adapts the injected model with answer-only QA supervision, while Recover merges the domain-adapted model with the base instruction model to recover general capabilities. Across Common Corpus (CC) and CCI, and across Llama, Phi, Qwen, and SmolLM model families, IAR improves the domain-primary domain-general frontier for retrieval-free document internalization. In the main comparison, IAR improves over Vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance across IFEval, MMLU, and MSBench. Extended CC baselines show that LoRA and FAPM can win individual general metrics, but among methods that also reach leading or near-leading domain internalization, IAR retains one of the strongest general profiles.
21 pages, 4 figures. Includes Supplementary Material Sections A--G. Qian Kou and Xiaofeng Shi contributed equally and are co-corresponding authors. Hua Zhou is the project leader
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2608.20281 [cs.CL]
(or arXiv:2608.20281v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2608.20281 arXiv-issued DOI via DataCite (pending registration)
Text integral sub licența deschisă a publicației · 256 cuvinte
Subiecte conexe
You can graft SDF changes from base models onto post-trained models
LessWrong · 6 oct. 2026
Building trust in AI agents: A production-ready evaluation framework with Amazon Bedrock AgentCore
AWS · 16 oct. 2026
Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
Hugging Face · 18 aug. 2026
Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model
AMD · 24 iul. 2026