aiqbench analysis
This arXiv preprint proposes Self-Policy Distillation (SPD), a method that improves LLMs by fine-tuning on their own outputs after projecting activations into a low-rank subspace derived from gradients on correctness-defining tokens. The authors report gains of up to 13% over prior self-distillation methods and up to 16.3% over pre-trained baselines across code, math, and multiple-choice QA, along with a 15.7% out-of-domain advantage. These are self-reported results on benchmarks, so readers should look for independent replication, but the approach of reducing reliance on external verification signals is worth watching for practitioners training models without costly labeled data.
Self-Policy Distillation via Capability-Selective Subspace Projection
arXiv:2605.22675v2 Announce Type: replace Abstract: Self-distillation bootstraps large language models (LLMs) by training on their own generations. However, existing methods either rely on external signals to curate self-generated outputs (e.g., correctness filtering, execution feedback, and reward search), which are costly and unavailable for the best-performing frontier models, or skip curation entirely and train on all raw outputs, an approach that is often domain-specific and hard to generalize. Both also share a deeper weakness that self-generated outputs entangle task-relevant capability with others, such as stylistic patterns, formatting artifacts, and model-specific errors, diluting the signal for the specific capability one aims to improve. In this paper, we propose Self-Policy Distillation (SPD), which achieves generalizable, capability selective with only minimal external signal (<5% labeled data in each domain). Specifically, SPD extracts a low-rank capability subspace from the model's own gradients on correctness-defining tokens, projects key-value (KV) activations into this subspace during self-generation, and fine-tunes on the resulting raw outputs with standard next-token prediction loss.Through extensive experiments across code generation, mathematical reasoning, and multiple-choice QA, we show that SPD achieves up to 13% improvement over state-of-the-art self-distillation methods and up to 16.3% improvement over pre-trained baselines. Notably,SPD demonstrates superior generalizability, achieving 15.7% better performance under out-of-domain generalization settings.
Abstract:Self-distillation bootstraps large language models (LLMs) by training on their own generations. However, existing methods either rely on external signals to curate self-generated outputs (e.g., correctness filtering, execution feedback, and reward search), which are costly and unavailable for the best-performing frontier models, or skip curation entirely and train on all raw outputs, an approach that is often domain-specific and hard to generalize. Both also share a deeper weakness that self-generated outputs entangle task-relevant capability with others, such as stylistic patterns, formatting artifacts, and model-specific errors, diluting the signal for the specific capability one aims to improve. In this paper, we propose Self-Policy Distillation (SPD), which achieves generalizable, capability selective with only minimal external signal (<5% labeled data in each domain). Specifically, SPD extracts a low-rank capability subspace from the model's own gradients on correctness-defining tokens, projects key-value (KV) activations into this subspace during self-generation, and fine-tunes on the resulting raw outputs with standard next-token prediction this http URL extensive experiments across code generation, mathematical reasoning, and multiple-choice QA, we show that SPD achieves up to 13% improvement over state-of-the-art self-distillation methods and up to 16.3% improvement over pre-trained baselines. Notably,SPD demonstrates superior generalizability, achieving 15.7% better performance under out-of-domain generalization settings.
Computation and Language (cs.CL)
arXiv:2605.22675 [cs.CL]
(or arXiv:2605.22675v2 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.22675 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 294 words
Related coverage
Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
LessWrong · Oct 16, 2026
[AINews] Meta Muse Wonder & Wonder Full: MSL’s Claude 5.5-tier Frontier Model
Latent Space · Oct 13, 2026
[AINews] The Future of Latent Space
Latent Space · Sep 25, 2026
[AINews] Opus 5.5 is good at explainer videos
Latent Space · Sep 29, 2026