Models in the Same Family are NOT Trust-Equivalent

arXiv:2508.13533v2 Announce Type: replace Abstract: Within a model family, a smaller variant is often deployed as a drop-in replacement for a larger one when their performance is similar. However, performance alone does not tell the full story. We propose a framework to evaluate trust-equivalence between a larger model and a smaller one in the same family along two dimensions. The first is attribution alignment: do both models base their predictions on the same input features? The second is calibration similarity: do both models share the same relationship between confidence and accuracy? We evaluate the Llama-2 family on two text classification tasks: Natural Language Inference and Paraphrase Identification. Attribution alignment is measured using two well-known methods: LIME and SHAP. Agreement between model pairs is quantified via the Jaccard coefficient over top-K attributed features. We observe that attribution alignment between models is generally low, indicating that smaller and larger models base their predictions on different input features. Calibration similarity is assessed using ECE, MCE, Brier Score, and Reliability Diagrams. Calibration profiles differ substantially across model sizes. There is no consistent relationship between model size and calibration quality. We have additionally verified these trends on two encoder-only families: BERT and Vision Transformer. The results are consistent with those reported here. Our experimental results show that replacing a larger model with a smaller one from the same family is a multidimensional decision that requires consideration beyond performance measures alone. Trust-equivalence must be assessed explicitly. It cannot be assumed from performance alone.

Abstract:Within a model family, a smaller variant is often deployed as a drop-in replacement for a larger one when their performance is similar. However, performance alone does not tell the full story. We propose a framework to evaluate trust-equivalence between a larger model and a smaller one in the same family along two dimensions. The first is attribution alignment: do both models base their predictions on the same input features? The second is calibration similarity: do both models share the same relationship between confidence and accuracy? We evaluate the Llama-2 family on two text classification tasks: Natural Language Inference and Paraphrase Identification. Attribution alignment is measured using two well-known methods: LIME and SHAP. Agreement between model pairs is quantified via the Jaccard coefficient over top-K attributed features. We observe that attribution alignment between models is generally low, indicating that smaller and larger models base their predictions on different input features. Calibration similarity is assessed using ECE, MCE, Brier Score, and Reliability Diagrams. Calibration profiles differ substantially across model sizes. There is no consistent relationship between model size and calibration quality. We have additionally verified these trends on two encoder-only families: BERT and Vision Transformer. The results are consistent with those reported here. Our experimental results show that replacing a larger model with a smaller one from the same family is a multidimensional decision that requires consideration beyond performance measures alone. Trust-equivalence must be assessed explicitly. It cannot be assumed from performance alone.

Computation and Language (cs.CL); Machine Learning (cs.LG)

arXiv:2508.13533 [cs.CL]

(or arXiv:2508.13533v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2508.13533 Focus to learn more arXiv-issued DOI via DataCite

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

analiza aiqbench

Models in the Same Family are NOT Trust-Equivalent

Subiecte conexe