analiza aiqbench
This research investigates the feature-level mechanisms of knowledge distillation, revealing how student models prune low-frequency features to rely on reusable representations akin to "dark knowledge." The authors introduce Confusion Distillation (CD), a teacher-free self-distillation technique that leverages internal confusion patterns as dynamic targets, achieving performance gains over existing methods like CS-KD. This work advances the theoretical understanding of model compression and offers a computationally efficient alternative for training compact neural networks.
What Do Students Learn? A Feature-Level Analysis of Dark Knowledge
arXiv:2606.03052v2 Announce Type: replace Abstract: Knowledge Distillation (KD) is a powerful tool for model compression, yet the precise mechanisms by which student models acquire feature representations remain underexplored. In this work, we analyze student feature learning using the Interaction Tensor framework. Our analysis reveals that effective KD acts as a regularizer that prunes low-frequency, sample-specific features, encouraging the student to rely on a compact set of highly reusable features. Crucially, we observe that the dataset-level confusion matrix contains structural information analogous to the teacher's "Dark Knowledge." Leveraging this insight, we propose Confusion Distillation (CD), a teacher-free self-distillation method that utilizes the model's own evolving )confusion patterns as dynamic soft targets. CD achieves competitive performance on ResNet-34 and ResNet-50 for CIFAR-100, outperforming existing self-distillation methods like CS-KD and PS-KD by 1.2% while offering a computationally efficient alternative to standard KD.
Title:What Do Students Learn? A Feature-Level Analysis of Dark Knowledge
Abstract:Knowledge Distillation (KD) is a powerful tool for model compression, yet the precise mechanisms by which student models acquire feature representations remain underexplored. In this work, we analyze student feature learning using the Interaction Tensor framework. Our analysis reveals that effective KD acts as a regularizer that prunes low-frequency, sample-specific features, encouraging the student to rely on a compact set of highly reusable features. Crucially, we observe that the dataset-level confusion matrix contains structural information analogous to the teacher's "Dark Knowledge." Leveraging this insight, we propose Confusion Distillation (CD), a teacher-free self-distillation method that utilizes the model's own evolving )confusion patterns as dynamic soft targets. CD achieves competitive performance on ResNet-34 and ResNet-50 for CIFAR-100, outperforming existing self-distillation methods like CS-KD and PS-KD by 1.2% while offering a computationally efficient alternative to standard KD.
Machine Learning (cs.LG)
arXiv:2606.03052 [cs.LG]
(or arXiv:2606.03052v2 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2606.03052 Focus to learn more arXiv-issued DOI via DataCite
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Text integral sub licența deschisă a publicației · 236 cuvinte
Subiecte conexe
What Do Students Learn? A Feature-Level Analysis of Dark Knowledge
arXiv · 3 iun. 2026
Links #5: 2026/07
LessWrong · 31 iul. 2026
A Chinese Perspective on “On Social Reality in China”
LessWrong · 15 oct. 2026
Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition
arXiv · 14 sept. 2026