aiqbench analysis
This research introduces AFCS, a novel synthetic data pipeline that enforces the inclusion of fine-grained atomic facts during language model pretraining to enhance factual accuracy. By leveraging structure-aware extraction and source-grounded verification, the method demonstrates measurable improvements on QA benchmarks compared to existing chunk-level synthesis techniques. The approach addresses a critical gap in current training strategies where secondary entities and precise temporal attributes are often lost in rewritten corpora.
AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining
arXiv:2610.20086v1 Announce Type: new Abstract: Synthetic data is increasingly used in language-model pretraining. Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usually take a source passage or document segment as input and generate rewritten or QA-style samples from that text. This can improve surface form and prompt diversity, but it does not explicitly require fine-grained source facts -- especially secondary entities, list/table entries, and precise temporal attributes -- to appear in the synthesized corpus. We propose AFCS (Atomic-Fact-Conditioned Synthesis), a pipeline that first extracts source-supported atomic facts and then uses each retained fact to drive generation. AFCS uses structure-aware extraction over the document tree, source-grounded verification, local context expansion, and multi-format generation. In continued-pretraining experiments with OLMo-3-7B and a 32B base on the same FineWiki source corpus, AFCS outperforms WRAP and Nemotron-CC on SimpleQA, SimpleQA Verified, PopQA, and HotpotQA. Granularity ablations show that fact-level synthesis improves over chunk-level, paragraph-level, and entity-level synthesis; and a 100B-token mid-training integration improves factual benchmarks and raises the 12-task general benchmark average by +1.09.
Abstract:Synthetic data is increasingly used in language-model pretraining. Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usually take a source passage or document segment as input and generate rewritten or QA-style samples from that text. This can improve surface form and prompt diversity, but it does not explicitly require fine-grained source facts -- especially secondary entities, list/table entries, and precise temporal attributes -- to appear in the synthesized corpus. We propose AFCS (Atomic-Fact-Conditioned Synthesis), a pipeline that first extracts source-supported atomic facts and then uses each retained fact to drive generation. AFCS uses structure-aware extraction over the document tree, source-grounded verification, local context expansion, and multi-format generation. In continued-pretraining experiments with OLMo-3-7B and a 32B base on the same FineWiki source corpus, AFCS outperforms WRAP and Nemotron-CC on SimpleQA, SimpleQA Verified, PopQA, and HotpotQA. Granularity ablations show that fact-level synthesis improves over chunk-level, paragraph-level, and entity-level synthesis; and a 100B-token mid-training integration improves factual benchmarks and raises the 12-task general benchmark average by +1.09.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv:2610.20086 [cs.CL]
(or arXiv:2610.20086v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.20086 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 262 words
Related coverage
What We Learned Trying to Catch AI Liars: An Aletheia's Quest Retrospective
EleutherAI · Aug 25, 2026
Probing Factual Knowledge Transfer with Training Data Interventions
arXiv · Sep 2, 2026
RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection
arXiv · Jul 29, 2026
State of the Field: AI for Epistemics and Coordination
LessWrong · Oct 2, 2026