aiqbench analysis
This study reframes mixup-based knowledge distillation as a mechanism that enables students to acquire superior linearity and calibration independent of their teachers, even when exposed to out-of-distribution vicinal inputs. By decoupling accuracy transfer from uncertainty estimation, the approach significantly reduces overconfidence and improves reliability across standard benchmarks like CIFAR and ImageNet. For practitioners, this offers a robust pathway to enhance model trustworthiness without relying on standard teacher-student imitation.
Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions
arXiv:2606.12171v1 Announce Type: cross Abstract: Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs. Their interaction, however, remains poorly understood, particularly when mixup is applied only during student training. In this setting, the teacher is queried on inputs drawn from a vicinal distribution it never saw during training, a controlled mismatch whose effect on knowledge transfer has not been characterised. We show that this mismatch causes the teacher's supervisory signal to be dominated by distributional confusion rather than inter-class structure. Despite it, the student does not merely imitate the teacher: it independently acquires greater linearity in the vicinal region, a structural property that the teacher lacks, and goes beyond dark-knowledge transfer. KD with mixup consistently improves student accuracy and reduces overconfidence by an order of magnitude relative to the baseline, across CIFAR and ImageNet with varying-capacity teachers. Crucially, calibration propagates from teacher to student independently of accuracy transfer, and temperature scaling governs a measurable accuracy-calibration trade-off that becomes more pronounced under vicinal training. These results reframe mixup distillation not as a degraded version of standard KD, but as a richer transfer channel that simultaneously shapes discriminative performance, uncertainty estimation, and representational geometry.
Abstract:Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs. Their interaction, however, remains poorly understood, particularly when mixup is applied only during student training. In this setting, the teacher is queried on inputs drawn from a vicinal distribution it never saw during training, a controlled mismatch whose effect on knowledge transfer has not been characterised. We show that this mismatch causes the teacher's supervisory signal to be dominated by distributional confusion rather than inter-class structure. Despite it, the student does not merely imitate the teacher: it independently acquires greater linearity in the vicinal region, a structural property that the teacher lacks, and goes beyond dark-knowledge transfer. KD with mixup consistently improves student accuracy and reduces overconfidence by an order of magnitude relative to the baseline, across CIFAR and ImageNet with varying-capacity teachers. Crucially, calibration propagates from teacher to student independently of accuracy transfer, and temperature scaling governs a measurable accuracy-calibration trade-off that becomes more pronounced under vicinal training. These results reframe mixup distillation not as a degraded version of standard KD, but as a richer transfer channel that simultaneously shapes discriminative performance, uncertainty estimation, and representational geometry.
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
arXiv:2606.12171 [cs.CV]
(or arXiv:2606.12171v1 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2606.12171 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 309 words
Related coverage
Longitudinal Risk Prediction in Mammography with Privileged History Distillation
arXiv · Sep 11, 2026
Sparse Regression Distilled from a Single Robust Fit
arXiv · Sep 22, 2026
Prediction-Only Distillation in Linear and Logistic Regression
arXiv · Jul 20, 2026
TAM: Task-Aware Memory Distillation for Efficient Spatiotemporal Prediction
arXiv · Oct 10, 2026