aiqbench analysis
This research introduces Inverse Distillation Unlearning (IDU), a novel framework that simultaneously compresses multi-step diffusion models into efficient one-step generators while effectively removing specific training data subsets. By formulating distillation as a min-max objective over data distributions, the method achieves unlearning without requiring access to retained training examples or additional classifiers. The approach successfully reduces the generation frequency of forgotten classes on standard benchmarks like MNIST and CIFAR-10 while maintaining high-quality output for retained data.
Data Unlearning via Inverse Distillation
arXiv:2609.36099v1 Announce Type: new Abstract: Multi-step matching models, including flow and diffusion models, produce high-quality outputs but incur substantial inference costs and may reproduce unwanted components of their training datasets. We introduce Inverse Distillation Unlearning (IDU), a unified framework that simultaneously distills a teacher multi-step matching model into an efficient one-step student generator and suppresses outputs corresponding to a designated training subset. We first formulate distillation as a min-max objective over a data distribution and then represent this distribution as a mixture of the forget-set and the generated distributions. This allows us to compare this mixture with the teacher's training distribution and recover only the retained data at the optimum. Our method requires only a pretrained full-data teacher and data from the forget set, without access to retained training examples, extra feature extractors or classifiers. Extensive experiments on MNIST and CIFAR-10 datasets under flow-matching and score-based diffusion settings demonstrate that IDU substantially reduces the generation frequency of forgotten classes while preserving generation quality on the retained classes. To the best of our knowledge, IDU is the first unified framework for simultaneous unlearning and distillation in unconditional flow-matching and score-based models.
Abstract:Multi-step matching models, including flow and diffusion models, produce high-quality outputs but incur substantial inference costs and may reproduce unwanted components of their training datasets. We introduce Inverse Distillation Unlearning (IDU), a unified framework that simultaneously distills a teacher multi-step matching model into an efficient one-step student generator and suppresses outputs corresponding to a designated training subset. We first formulate distillation as a min-max objective over a data distribution and then represent this distribution as a mixture of the forget-set and the generated distributions. This allows us to compare this mixture with the teacher's training distribution and recover only the retained data at the optimum. Our method requires only a pretrained full-data teacher and data from the forget set, without access to retained training examples, extra feature extractors or classifiers. Extensive experiments on MNIST and CIFAR-10 datasets under flow-matching and score-based diffusion settings demonstrate that IDU substantially reduces the generation frequency of forgotten classes while preserving generation quality on the retained classes. To the best of our knowledge, IDU is the first unified framework for simultaneous unlearning and distillation in unconditional flow-matching and score-based models.
Machine Learning (cs.LG)
arXiv:2609.36099 [cs.LG]
(or arXiv:2609.36099v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.36099 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 279 words
Related coverage
Inverse Reasoning Models for Efficient Post-Training and Verification from Answer-Only Data
arXiv · Oct 15, 2026
SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion
arXiv · Jun 6, 2026
Enabling Preference-driven Unlearning in Few-step Distilled Text-to-Image Diffusion Models
arXiv · Oct 9, 2026
Overclocking Electrostatic Generative Models
arXiv · Jun 5, 2026