Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies

Abstract:The ranking of recommendation algorithms is a challenging problem since model performance is sensitive to dataset characteristics such as sparsity, sequential structure, and scale. This drives a demand for a proper methodology for fair comparison between algorithms. Naive aggregation of performance metrics (e.g., averaging NDCG over benchmarks) can yield misleading rankings, undermining practical selection. To address this problem, we introduce a novel, data-driven ranking methodology based on Bradley-Terry (BT) model. We demonstrate that the obtained ranking depends on key dataset statistics. Additionally, we propose a novel metric for evaluating ranking consistency and demonstrate robustness of our ranking to incomplete data. Finally, we introduce a dataset-specific methodology for ranking algorithms on unseen datasets without running the models, relying on extensions of the Bradley-Terry framework, including BT trees and BT models with covariates.

Information Retrieval (cs.IR); Machine Learning (cs.LG); Machine Learning (stat.ML)

arXiv:2606.07492 [cs.IR]

(or arXiv:2606.07492v1 [cs.IR] for this version)

https://doi.org/10.48550/arXiv.2606.07492 arXiv-issued DOI via DataCite (pending registration)

https://doi.org/10.1145/3770855.3817890 DOI(s) linking to related resources