aiqbench analysis
This research reveals that standard global fidelity scores for surrogate models are insufficient, as agreement often breaks down specifically near a teacher model's decision boundaries. The study demonstrates that evaluating local stability in these regions provides critical insights into how surrogates respond to distribution shifts, particularly when using independently trained deep teachers with unstable boundaries. For practitioners relying on distillation or approximation strategies, this highlights the necessity of boundary-aware metrics to ensure robust model performance.
"Where Can I Trust You?": Boundary-Aware Evaluation of Surrogate Fidelity
arXiv:2609.32230v1 Announce Type: new Abstract: Surrogate models are commonly evaluated by how often they agree with their teacher model over an evaluation set. Local variation in this agreement is well known, but its structure and consequences are less clear. We ask whether disagreement is systematically concentrated near the teacher's decision boundary and whether retaining that structure provides information beyond a single global score. Across several datasets and surrogate model classes, we find substantially lower fidelity near teacher decision boundaries under two different methods of identifying near-boundary examples. Moreover, conditioning agreement on confidence-defined regions improves prediction of teacher--surrogate agreement when evaluation-set composition changes, relative to the global score alone. Yet surrogates that agree equally well with the teacher both globally and near the decision boundary can respond very differently to changes selected using the surrogate itself. Finally, we show that independently trained deep teachers can agree on most predictions while identifying different examples as lying near their decision boundaries, complicating the use of those boundaries as stable reference regions for evaluating surrogates. Together, these results show that surrogate fidelity depends not only on how often a surrogate agrees with its teacher, but also on where that agreement holds and, for deep models, how stable the teacher's decision boundary is across training runs.
Title:"Where Can I Trust You?": Boundary-Aware Evaluation of Surrogate Fidelity
Abstract:Surrogate models are commonly evaluated by how often they agree with their teacher model over an evaluation set. Local variation in this agreement is well known, but its structure and consequences are less clear. We ask whether disagreement is systematically concentrated near the teacher's decision boundary and whether retaining that structure provides information beyond a single global score. Across several datasets and surrogate model classes, we find substantially lower fidelity near teacher decision boundaries under two different methods of identifying near-boundary examples. Moreover, conditioning agreement on confidence-defined regions improves prediction of teacher--surrogate agreement when evaluation-set composition changes, relative to the global score alone. Yet surrogates that agree equally well with the teacher both globally and near the decision boundary can respond very differently to changes selected using the surrogate itself. Finally, we show that independently trained deep teachers can agree on most predictions while identifying different examples as lying near their decision boundaries, complicating the use of those boundaries as stable reference regions for evaluating surrogates. Together, these results show that surrogate fidelity depends not only on how often a surrogate agrees with its teacher, but also on where that agreement holds and, for deep models, how stable the teacher's decision boundary is across training runs.
Machine Learning (cs.LG)
arXiv:2609.32230 [cs.LG]
(or arXiv:2609.32230v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2609.32230 Focus to learn more arXiv-issued DOI via DataCite (pending registration)
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Full text under the publisher's open licence · 310 words
Related coverage
AI and Quantum Computing Briefing
livetradingnews.com · Aug 4, 2026
BAT-NO: A Boundary-Condition-Aware Transformer Neural Operator for Crashworthiness Prediction of Vehicle Components
arXiv · Oct 6, 2026
DiffGA: Steering Frozen Diffusion Models for Materials Design via Evolutionary Search
arXiv · Oct 14, 2026
Robust Belief-State Policy Learning for Quantum Network Routing Under Decoherence and Time-Varying Conditions
arXiv · Jul 23, 2026