Robust and sparse logistic regression for high-dimensional data: A DPD-ENP hybrid model

Abstract

A popular statistical technique for modeling binary response variables is logistic regression. Nevertheless, the performance of standard logistic regression may be affected by the sensitivity of its maximum likelihood estimation to correlated predictor variables and outliers. In addition, traditional estimation techniques can occasionally become too complicated when working with high-dimensional datasets. In order to overcome these constraints, we provide a sparse and robust logistic regression model that makes use of the elastic net penalty as a regularizer that induces sparsity and density power divergence for robustness. Our method makes use of k-fold cross-validation to guaranty model stability and generalizability along with the majorization-minimization algorithm for effective parameter estimation. The efficacy of the suggested strategy for managing outliers in high dimensions is demonstrated through the execution of simulated datasets and a real-time example using the breast cancer data set.

Keywords

• Density power divergence
• elastic net penalty
• k-fold cross-validation
• logistic regression
• MM algorithm
• robust
• sparse

References

1. [1] B. Akturk, U. Beyaztas, H.L. Shang and A. Mandal, Robust functional logistic regression. Adv. Data Anal. Classif. 18 (1), 125, 2024.
2. [2] Z. Y. Algamal and M. H. Lee, High dimensional logistic regression model using adjusted elastic net penalty, Pak. J. Stat. Oper. Res. 11 (4), 667676, 2015.
3. [3] A. Basu, I. R. Harris, N. L. Hjort and M. C. Jones, Robust and efficient estimation by minimising a density power divergence, Biometrika 85 (3), 549559, 1998.
4. [4] A. M. Bianco and V. J. Yohai, Robust estimation in the logistic regression model, Robust Stat., Data Anal. Comput. Intensive Methods, Springer 109, 1734, 1996.
5. [5] E. Cantoni and E. Ronchetti, Robust inference for generalized linear models, J. Am. Stat. Assoc. 96 (455), 10221030, 2001.
6. [6] R. J. Carroll and S. Pederson, On robustness in the logistic regression model, J. R. Stat. Soc. Ser. B (Methodol.) 55 (3), 693706, 1993.
7. [7] R. D. Cook and S. Weisberg, Residuals and Influence in Regression, Chapman and Hall, New York, 1982. Available at: https://conservancy.umn.edu/handle/11299/37076
8. [8] D. Cornilly, L. Tubex, S. Van Aelst and T. Verdonck, Robust and sparse logistic regression, Adv. Data Anal. Classif. 18 (3), 663679, 2024.

9. [9] A. Felipe, M. Jaenada, P. Miranda and L. Pardo, Robust parameter estimation of the log-logistic distribution based on density power divergence estimators, arXiv:2312.02662, 2023.
10. [10] J. Feng, H. Xu, S. Mannor and S. Yan, Robust logistic regression and classification, Adv. Neural Inf. Process. Syst. 1, 253261, 2014.
11. [11] Greeshmagiri and T. Palanisamy, A hard re-descending hybrid robust regression estimation technique using direct weights, Hacet. J. Math. Stat. 53 (5), 14381452, 2024. https://doi.org/10.15672/hujms.1383910
12. [12] Greeshmagiri and T. Palanisamy, Constrained robust regression of interval valued data, J. Stat. Theory Pract. 19, 2025. https://doi.org/10.1007/s42519-025-00450-6
13. [13] D. Hou, W. Zhou, Q. Zhang, K. Zhang and J. Fang, A comparative study of different variable selection methods based on numerical simulation and empirical analysis, PeerJ Comput. Sci. 9, e1522, 2023.
14. [14] D. E. Jennings, Outliers and residual distributions in logistic regression, J. Am. Stat. Assoc. 81 (396), 987990, 1986.
15. [15] P. Komarek, Logistic Regression for Data Mining and High-dimensional Classification, Carnegie Mellon Univ., 2004.
16. [16] F. S. Kurnaz, I. Hoffmann and P. Filzmoser, Robust and sparse estimation methods for high-dimensional linear and logistic regression, Chemom. Intell. Lab. Syst. 172, 211222, 2018.
17. [17] J. M. Landwehr, D. Pregibon and A. C. Shoemaker, Graphical methods for assessing logistic regression models, J. Am. Stat. Assoc. 79 (385), 6171, 1984.
18. [18] W. Li, C. Li and L. Jiang, Learning from crowds with robust logistic regression, Inf. Sci. 639, 119010, 2023. https://doi.org/10.1016/j.ins.2023.119010
19. [19] D. Pregibon, Logistic regression diagnostics, Ann. Stat. 9 (4), 705724, 1981.
20. [20] M. Riani, A. C. Atkinson, A. Corbellini and D. Perrot, Robust regression with density power divergence: theory, comparisons, and data analysis, Entropy 22 (4), 399, 2020.
21. [21] S. Roy, A. Basu and A. Ghosh, Robust principal component analysis using density power divergence, arXiv:2309.13531, 2023. https://arxiv.org/abs/2309.13531
22. [22] G. Saraceno, A. Ghosh, A. Basu and C. Agostinelli, Robust estimation of fixed effect parameters and variances of linear mixed models: the minimum density power divergence approach, AStA Adv. Stat. Anal. 108 (1), 127157, 2024.
23. [23] B. Shin and S. Lee, Robust logistic regression with shift parameter estimation, J. Stat. Comput. Simul. 93 (15), 26252641, 2023. https://doi.org/10.1080/00949655.2023.2201008
24. [24] Z. Song, L. Wang, X. Xu and W. Zhao, Doubly robust logistic regression for image classification, Appl. Math. Model. 123, 430446, 2023. https://doi.org/10.1016/j.apm.2023.06.039
25. [25] T. Zhou, D. Tao and X. Wu, Manifold elastic net: a unified framework for sparse dimension reduction, Data Min. Knowl. Discov. 22, 340371, 2011.
26. [26] H. Zou and T. Hastie, Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B (Stat. Methodol.) 67 (2), 301320, 2005.