aiqbench analysis
The paper studies catastrophic forgetting in LLM agents that learn new procedural skills over time, and identifies two distinct failure stages: models first lose the ability to select an old skill from a raw request, then lose the ability to execute it even when given the correct skill name. Its proposed Factorized Skill Memory uses frozen per-skill LoRA adapters plus a trainable router, and reports 65.95% exact match on a five-stage, 100-skill TaskBench benchmark without replaying historical data. The results are benchmark-specific and the replay-free comparison is the key claim, so readers should check how well the routing approach holds up on less structured skill sets.
When Skills Remain But Access Fails: Factorized Skill Memory for Continual Skill Internalization
arXiv:2610.15492v1 Announce Type: new Abstract: Skill internalization aims to convert externally specified procedural skills into persistent model capabilities, allowing language-model agents to execute them without retrieving or reinjecting skill descriptions at inference time. Yet when new skills are learned continually, whether previously internalized skills remain usable is still poorly understood. We study forgetting in continual skill internalization and uncover a clear hierarchical structure. Early in subsequent training, a model may lose the ability to invoke an old skill from the raw request, while supplying only the correct canonical skill name substantially restores the learned behavior; with further optimization, this conditional execution also degrades. We call these two stages skill-addressing forgetting and skill-execution forgetting. Guided by this finding, we introduce Factorized Skill Memory (FSM), which separates continual skill memory into a plastic Address Memory and a stable Execution Memory. At each stage, FSM trains a new LoRA executor using only current-stage data, freezes it after acquisition, and appends it to Execution Memory; in parallel, Address Memory incrementally learns to map current requests to executors without replaying historical requests or execution trajectories. At inference time, Address Memory selects one executor from the raw request, and only that frozen executor performs a single language-model generation. On a five-stage benchmark built from 100 TaskBench skills, FSM achieves 65.95% mean exact match and 93.38% addressing accuracy across two task orders, improving over the strongest replay-free parameter-isolation baseline by 14.3 percentage points while remaining within 1.9 points of Replay, which retains historical training examples.
Abstract:Skill internalization aims to convert externally specified procedural skills into persistent model capabilities, allowing language-model agents to execute them without retrieving or reinjecting skill descriptions at inference time. Yet when new skills are learned continually, whether previously internalized skills remain usable is still poorly understood. We study forgetting in continual skill internalization and uncover a clear hierarchical structure. Early in subsequent training, a model may lose the ability to invoke an old skill from the raw request, while supplying only the correct canonical skill name substantially restores the learned behavior; with further optimization, this conditional execution also degrades. We call these two stages skill-addressing forgetting and skill-execution forgetting. Guided by this finding, we introduce Factorized Skill Memory (FSM), which separates continual skill memory into a plastic Address Memory and a stable Execution Memory. At each stage, FSM trains a new LoRA executor using only current-stage data, freezes it after acquisition, and appends it to Execution Memory; in parallel, Address Memory incrementally learns to map current requests to executors without replaying historical requests or execution trajectories. At inference time, Address Memory selects one executor from the raw request, and only that frozen executor performs a single language-model generation. On a five-stage benchmark built from 100 TaskBench skills, FSM achieves 65.95% mean exact match and 93.38% addressing accuracy across two task orders, improving over the strongest replay-free parameter-isolation baseline by 14.3 percentage points while remaining within 1.9 points of Replay, which retains historical training examples.
Artificial Intelligence (cs.AI)
arXiv:2610.15492 [cs.AI]
(or arXiv:2610.15492v1 [cs.AI] for this version)
https://doi.org/10.48550/arXiv.2610.15492 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 337 words
Related coverage
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
The J-Space Debate, Agent Swarms, and Pacing Frontier AI - Digital Minds Newsletter #4
LessWrong · Sep 18, 2026
Performance Profiling on AMD GPUs – Part 5: Profiling-Driven Kernel Optimization with an AI Code-Assist Tool
AMD · Jul 16, 2026
Game Design Creativity evals
LessWrong · Oct 12, 2026