Robust and sparse logistic regression for high-dimensional data: A DPD-ENP hybrid model

Year 2025, Volume: 54 Issue: 6, 2463 - 2482, 30.12.2025

K Arya , Palanisamy T

https://doi.org/10.15672/hujms.1727381

Abstract

A popular statistical technique for modeling binary response variables is logistic regression. Nevertheless, the performance of standard logistic regression may be affected by the sensitivity of its maximum likelihood estimation to correlated predictor variables and outliers. In addition, traditional estimation techniques can occasionally become too complicated when working with high-dimensional datasets. In order to overcome these constraints, we provide a sparse and robust logistic regression model that makes use of the elastic net penalty as a regularizer that induces sparsity and density power divergence for robustness. Our method makes use of k-fold cross-validation to guaranty model stability and generalizability along with the majorization-minimization algorithm for effective parameter estimation. The efficacy of the suggested strategy for managing outliers in high dimensions is demonstrated through the execution of simulated datasets and a real-time example using the breast cancer data set.

Keywords

Density power divergence , elastic net penalty , k-fold cross-validation , logistic regression , MM algorithm , robust , sparse

References

• [1] B. Akturk, U. Beyaztas, H.L. Shang and A. Mandal, Robust functional logistic regression. Adv. Data Anal. Classif. 18 (1), 125, 2024.
• [2] Z. Y. Algamal and M. H. Lee, High dimensional logistic regression model using adjusted elastic net penalty, Pak. J. Stat. Oper. Res. 11 (4), 667676, 2015.
• [3] A. Basu, I. R. Harris, N. L. Hjort and M. C. Jones, Robust and efficient estimation by minimising a density power divergence, Biometrika 85 (3), 549559, 1998.
• [4] A. M. Bianco and V. J. Yohai, Robust estimation in the logistic regression model, Robust Stat., Data Anal. Comput. Intensive Methods, Springer 109, 1734, 1996.
• [5] E. Cantoni and E. Ronchetti, Robust inference for generalized linear models, J. Am. Stat. Assoc. 96 (455), 10221030, 2001.
• [6] R. J. Carroll and S. Pederson, On robustness in the logistic regression model, J. R. Stat. Soc. Ser. B (Methodol.) 55 (3), 693706, 1993.
• [7] R. D. Cook and S. Weisberg, Residuals and Influence in Regression, Chapman and Hall, New York, 1982. Available at: https://conservancy.umn.edu/handle/11299/37076
• [8] D. Cornilly, L. Tubex, S. Van Aelst and T. Verdonck, Robust and sparse logistic regression, Adv. Data Anal. Classif. 18 (3), 663679, 2024.
• [9] A. Felipe, M. Jaenada, P. Miranda and L. Pardo, Robust parameter estimation of the log-logistic distribution based on density power divergence estimators, arXiv:2312.02662, 2023.
• [10] J. Feng, H. Xu, S. Mannor and S. Yan, Robust logistic regression and classification, Adv. Neural Inf. Process. Syst. 1, 253261, 2014.
• [11] Greeshmagiri and T. Palanisamy, A hard re-descending hybrid robust regression estimation technique using direct weights, Hacet. J. Math. Stat. 53 (5), 14381452, 2024. https://doi.org/10.15672/hujms.1383910
• [12] Greeshmagiri and T. Palanisamy, Constrained robust regression of interval valued data, J. Stat. Theory Pract. 19, 2025. https://doi.org/10.1007/s42519-025-00450-6
• [13] D. Hou, W. Zhou, Q. Zhang, K. Zhang and J. Fang, A comparative study of different variable selection methods based on numerical simulation and empirical analysis, PeerJ Comput. Sci. 9, e1522, 2023.
• [14] D. E. Jennings, Outliers and residual distributions in logistic regression, J. Am. Stat. Assoc. 81 (396), 987990, 1986.
• [15] P. Komarek, Logistic Regression for Data Mining and High-dimensional Classification, Carnegie Mellon Univ., 2004.
• [16] F. S. Kurnaz, I. Hoffmann and P. Filzmoser, Robust and sparse estimation methods for high-dimensional linear and logistic regression, Chemom. Intell. Lab. Syst. 172, 211222, 2018.
• [17] J. M. Landwehr, D. Pregibon and A. C. Shoemaker, Graphical methods for assessing logistic regression models, J. Am. Stat. Assoc. 79 (385), 6171, 1984.
• [18] W. Li, C. Li and L. Jiang, Learning from crowds with robust logistic regression, Inf. Sci. 639, 119010, 2023. https://doi.org/10.1016/j.ins.2023.119010
• [19] D. Pregibon, Logistic regression diagnostics, Ann. Stat. 9 (4), 705724, 1981.
• [20] M. Riani, A. C. Atkinson, A. Corbellini and D. Perrot, Robust regression with density power divergence: theory, comparisons, and data analysis, Entropy 22 (4), 399, 2020.
• [21] S. Roy, A. Basu and A. Ghosh, Robust principal component analysis using density power divergence, arXiv:2309.13531, 2023. https://arxiv.org/abs/2309.13531
• [22] G. Saraceno, A. Ghosh, A. Basu and C. Agostinelli, Robust estimation of fixed effect parameters and variances of linear mixed models: the minimum density power divergence approach, AStA Adv. Stat. Anal. 108 (1), 127157, 2024.
• [23] B. Shin and S. Lee, Robust logistic regression with shift parameter estimation, J. Stat. Comput. Simul. 93 (15), 26252641, 2023. https://doi.org/10.1080/00949655.2023.2201008
• [24] Z. Song, L. Wang, X. Xu and W. Zhao, Doubly robust logistic regression for image classification, Appl. Math. Model. 123, 430446, 2023. https://doi.org/10.1016/j.apm.2023.06.039
• [25] T. Zhou, D. Tao and X. Wu, Manifold elastic net: a unified framework for sparse dimension reduction, Data Min. Knowl. Discov. 22, 340371, 2011.
• [26] H. Zou and T. Hastie, Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B (Stat. Methodol.) 67 (2), 301320, 2005.