aiqbench analysis
Cortex introduces a novel framework for memory consolidation in large language models, effectively distilling context-dependent gains into stable network updates without degrading original capabilities. By leveraging prospective queries to anticipate context usage, this method achieves high retention of factual edits across diverse model architectures, addressing the critical challenge of maintaining performance after context removal. This research advances the field by unifying skill learning and behavioral steering under a single objective function inspired by biological memory systems.
Cortex: Consolidating Context into Parametric Memory
arXiv:2610.20399v1 Announce Type: new Abstract: Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B--120B dense and mixture-of-experts backbones, including both text-only and multimodal models, Cortex lies on the empirical consolidation--retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.
Abstract:Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B--120B dense and mixture-of-experts backbones, including both text-only and multimodal models, Cortex lies on the empirical consolidation--retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
arXiv:2610.20399 [cs.CL]
(or arXiv:2610.20399v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2610.20399 arXiv-issued DOI via DataCite (pending registration)
Full text under the publisher's open licence · 166 words
Related coverage
Benchtalks #3: We taught AI everything except how to learn
Snorkel AI Blog · Jun 25, 2026
What's Continual Learning, and Why Might We Expect To See It In Advanced LLM Agents?
LessWrong · Jun 12, 2026
A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
arXiv · Aug 6, 2026
PEAM: Parametric Embodied Agent Memory through Contrastive Internalization of Experience in Minecraft
arXiv · Jun 3, 2026