CALIBRATED POPULARITY RE-RANKING WITH ALTERNATIVE DIVERGENCE MEASURES FOR POPULARITY BIAS MITIGATION

Year 2025, Volume: 26 Issue: 3, 260 - 278, 25.09.2025

Emre Yalçın

https://doi.org/10.18038/estubtda.1678307

Abstract

Popularity bias significantly limits the effectiveness of recommender systems by disproportionately favoring popular items and reducing exposure to diverse, less-known content. This bias negatively impacts personalization and marginalizes niche users and item providers. To address this challenge, calibrated recommendation methods have gained attention, notably the Calibrated Popularity (CP) approach, due to its simplicity, effectiveness, and model-agnostic nature. Originally, CP employs Jensen–Shannon divergence (JSD) to align the popularity distribution of recommended items with users’ historical interaction patterns. However, the choice of divergence measure substantially impacts calibration effectiveness and recommendation diversity. In this study, we systematically explore several alternative divergence measures, including Chi-Square, Wasserstein, Kullback–Leibler, Hellinger, Total Variation, Bhattacharyya, Cosine, and Renyi divergences, within the CP framework. Additionally, we propose a novel divergence-independent evaluation metric, namely Overall Similarity Error, enabling consistent assessment of calibration quality across divergence measures. Experimental results on two benchmark datasets using two collaborative filtering algorithms highlighted critical insights. More aggressive divergences, particularly Chi-Square, significantly enhanced calibration quality, reduced popularity bias, and increased recommendation diversity, albeit with a modest reduction in accuracy. In contrast, smoother divergences, such as JSD, maintained higher accuracy but provided limited improvements in reducing popularity bias. Also, the performed group-based analysis categorizing users into mainstream, balanced, and niche segments based on their historical popularity preferences revealed distinct patterns: balanced users typically achieved higher accuracy due to their evenly distributed preferences; mainstream users showed superior calibration results benefiting from robust signals of popular items; niche users obtained more diverse and personalized recommendations, clearly benefiting from aggressive divergence measures. These results underscore the complexity of addressing popularity bias and highlight the importance of adopting adaptive, user-aware calibration strategies to effectively balance accuracy, diversity, and fairness in recommender systems.

Keywords

Divergence measures , Popularity bias , Calibrated recommendation , Collaborative filtering , Personalization.

References

• [1] Zangerle E, Bauer C. Evaluating recommender systems: survey and framework. ACM Comput Surv 2022; 55(8): 1-38.
• [2] Ahanger AB, Aalam SW, Bhat MR, Assad A. Popularity bias in recommender systems-a review. In: Int Conf Emerging Technol Comput Eng; February 2022; Cham, Switzerland. Cham: Springer. pp. 431-444.
• [3] Abdollahpouri H, Mansoury M, Burke R, Mobasher B. The unfairness of popularity bias in recommendation. arXiv preprint arXiv:1907.13286.
• [4] Mansoury M, Abdollahpouri H, Pechenizkiy M, Mobasher B, Burke R. Feedback loop and bias amplification in recommender systems. In: 29th ACM Int Conf Inf Knowl Manag; 2020; New York, NY, USA. New York: ACM. pp. 2145-2148.
• [5] Steck H. Calibrated recommendations. In: 12th ACM Conf Recommender Syst; 2018; New York, NY, USA. New York: ACM. pp. 154-162.
• [6] Abdollahpouri H, Mansoury M, Burke R, Mobasher B, Malthouse E. User-centered evaluation of popularity bias in recommender systems. In: 29th ACM Conf User Modeling, Adaptation and Personalization; 2021; New York, NY, USA. New York: ACM. pp. 119-129.
• [7] Yalcin E, Bilge A. Evaluating unfairness of popularity bias in recommender systems: A comprehensive user-centric analysis. Inf Process Manag 2022; 59(6): 103100.
• [8] Pardo L. Statistical Inference Based on Divergence Measures. Boca Raton, FL, USA: Chapman and Hall/CRC, 2018.
• [9] Kaya M, Bridge D. A comparison of calibrated and intent-aware recommendations. In: 13th ACM Conf Recommender Syst; 2019; New York, NY, USA. New York: ACM. pp. 151-159.
• [10] Seymen S, Abdollahpouri H, Malthouse EC. A constrained optimization approach for calibrated recommendations. In: 15th ACM Conf Recommender Syst; 2021; New York, NY, USA. New York: ACM. pp. 607-612.
• [11] da Silva DC, Manzato MG, Durão FA. Exploiting personalized calibration and metrics for fairness recommendation. Expert Syst Appl 2021; 181: 115112.
• [12] da Silva DC, Durão FA. Benchmarking fairness measures for calibrated recommendation systems on movies domain. Expert Syst Appl 2025; 269: 126380.
• [13] Cha SH. Comprehensive survey on distance/similarity measures between probability density functions. Int J Math Model Methods Appl Sci 2007; 1(2): 1.
• [14] Burke R. Multisided fairness for recommendation. arXiv preprint arXiv:1707.00093.
• [15] Deldjoo Y, Jannach D, Bellogin A, Difonzo A, Zanzonelli D. Fairness in recommender systems: research landscape and future directions. User Model User-Adap Interact 2024; 34(1): 59-108.
• [16] Vassøy B, Langseth H. Consumer-side fairness in recommender systems: a systematic survey of methods and evaluation. Artif Intell Rev 2024; 57(4): 101.
• [17] Yalcin E, Bilge A. Popularity bias in personality perspective: An analysis of how personality traits expose individuals to the unfair recommendation. Concurrency Comput Pract Exp 2023; 35(9): e7647.
• [18] Dunford R, Su Q, Tamang E, Wintour A. The pareto principle. Plymouth Stud Sci 2014; 7(1): 140-148.
• [19] Cui J, Tian Z, Zhong Z, Qi X, Yu B, Zhang H. Decoupled kullback-leibler divergence loss. Adv Neural Inf Process Syst 2024; 37: 74461-74486.
• [20] Feng W, Liu L, Liu T. On deterministically approximating total variation distance. In: ACM-SIAM Symp Discrete Algorithms (SODA); 2024; New York, NY, USA. New York: SIAM. pp. 1766-1791.
• [21] Nietert S, Goldfeld Z, Sadhu R, Kato K. Statistical, robustness, and computational guarantees for sliced wasserstein distances. Adv Neural Inf Process Syst 2022; 35: 28179-28193.
• [22] Li X, Liu Z, Han X, Liu N, Yuan W. An intuitionistic fuzzy version of hellinger distance measure and its application to decision-making process. Symmetry 2023; 15(2): 500.
• [23] Shen C, Panda S, Vogelstein JT. The chi-square test of distance correlation. J Comput Graph Stat 2022; 31(1): 254-262.
• [24] Huang Y, Xiao F, Cao Z, Lin CT. Higher order fractal belief Rényi divergence with its applications in pattern classification. IEEE Trans Pattern Anal Mach Intell 2023; 45(12): 14709-14726.
• [25] Baidari I, Honnikoll N. Bhattacharyya distance based concept drift detection method for evolving data stream. Expert Syst Appl 2021; 183: 115303.
• [26] Geroldinger A, Lusa L, Nold M, Heinze G. Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures—a simulation study. Diagn Progn Res 2023; 7(1): 9.
• [27] Yalcin E, Bilge A. Treating adverse effects of blockbuster bias on beyond-accuracy quality of personalized recommendations. Eng Sci Technol Int J 2022; 33: 101083.
• [28] Salah A, Rogovschi N, Nadif M. A dynamic collaborative filtering system via a weighted clustering approach. Neurocomputing 2016; 175: 206-215.
• [29] Liang D, Krishnan RG, Hoffman MD, Jebara T. Variational autoencoders for collaborative filtering. In: WWW 2018; 2018; Lyon, France. New York: ACM. pp. 689-698.
• [30] Salah A, Truong QT, Lauw HW. Cornac: A comparative framework for multimodal recommender systems. J Mach Learn Res 2020; 21(95): 1-5.