analiza aiqbench
This paper introduces a statistical framework for evaluating generative models by utilizing KL divergence to quantify uncertainty without relying on tuning parameters like kernel bandwidths. By extending the method to conditional models and leveraging Edgeworth expansions for limited data, it offers a more robust alternative to existing benchmark metrics with higher power in simulated settings. The work addresses a critical gap in principled evaluation methods essential for validating AI systems across image and text domains.
Statistical Inference for Generative Model Comparison
arXiv:2501.18897v4 Announce Type: replace-cross Abstract: Generative models have achieved remarkable success across a range of applications, yet their evaluation still lacks principled uncertainty quantification. In this paper, we develop a method for comparing how close different generative models are to the underlying distribution of test samples. Particularly, our approach employs the Kullback-Leibler (KL) divergence to measure the distance between a generative model and the unknown test distribution, as KL requires no tuning parameters such as the kernels used by RKHS-based distances, and is the only $f$-divergence that admits a crucial cancellation to enable the uncertainty quantification. Furthermore, we extend our method to comparing conditional generative models and leverage Edgeworth expansions to address limited-data settings. On simulated datasets with known ground truth, we show that our approach realizes effective coverage rates, and has higher power compared to kernel-based methods. When applied to generative models on image and text datasets, our procedure yields conclusions consistent with benchmark metrics but with statistical confidence.
Abstract:Generative models have achieved remarkable success across a range of applications, yet their evaluation still lacks principled uncertainty quantification. In this paper, we develop a method for comparing how close different generative models are to the underlying distribution of test samples. Particularly, our approach employs the Kullback-Leibler (KL) divergence to measure the distance between a generative model and the unknown test distribution, as KL requires no tuning parameters such as the kernels used by RKHS-based distances, and is the only $f$-divergence that admits a crucial cancellation to enable the uncertainty quantification. Furthermore, we extend our method to comparing conditional generative models and leverage Edgeworth expansions to address limited-data settings. On simulated datasets with known ground truth, we show that our approach realizes effective coverage rates, and has higher power compared to kernel-based methods. When applied to generative models on image and text datasets, our procedure yields conclusions consistent with benchmark metrics but with statistical confidence.
Machine Learning (stat.ML); Machine Learning (cs.LG)
arXiv:2501.18897 [stat.ML]
(or arXiv:2501.18897v4 [stat.ML] for this version)
Text integral sub licența deschisă a publicației · 169 cuvinte
Subiecte conexe
Inference meta-monitoring for Amazon SageMaker AI endpoints with Amazon Quick
AWS · 30 iul. 2026
[AINews] Reflection Beam - 501B-A23B American Open Model
Latent Space · 6 oct. 2026
Informe de riesgos de la IA de frontera (febrero–marzo de 2026)
METR · 19 mai 2026
Frontier AI Risk Report (February–March 2026)
METR · 19 mai 2026