Novel heteroscedastic robust ridge M-estimators for linear regression model

Abstract

Penalized regression techniques are widely used when the number of predictors is large and they are highly inter-correlated. Within penalized linear regression techniques, ridge regression is primarily used to obtain estimators that minimize the mean squared error while fitting the regression model. This research work addresses the issue of ridge estimation when multicollinearity, heteroscedasticity, and outliers exist within the predictors. For example, in a dataset that predicts livestock production based on factors such as animal population and feed quality, an outlier, such as an unusually high number of livestock in one region due to a large farm, could distort the regression results. To this end, we introduced a novel heteroscedastic ridge M-estimator approach in conjunction with the scaling factor method. This new approach improves the performance of ridge estimators compared to traditional techniques. To demonstrate its effectiveness, the proposed methodology is compared with several widely used and latest ridge estimators. Extensive simulations demonstrate the remarkable performance of the proposed approach, particularly in the presence of severe multicollinearity, heteroscedasticity, and outliers. Furthermore, real-world applications validate the potential of this methodology as a valuable tool for data analysis in scenarios characterized by collinear predictors and heteroscedastic errors.

Keywords

• Heteroscedastic ridge regression
• multicollinearity
• mean squared error
• ordinary least squares
• outlier
• ridge regression
• robust ridge regression.

References

1. [1] S. Mermi, Ö. Akkuş, A. Göktaş and N. Gündüz, A new robust ridge parameter estimator having no outlier and ensuring normality for linear regression model. J. Radiat. Res. Appl. Sci. 17 (1), 100788, 2024.
2. [2] A. Göktaş, Ö. Akkuş and A. Kuvat, A new robust ridge parameter estimator based on search method for linear regression model. J. Appl. Stat. 48 (13-15), 2457-2472, 2021.
3. [3] S. Ali, H. Khan, I. Shah, M. M. Butt and M. Suhail, A comparison of some new and old robust ridge regression estimators. Commun. Stat. Simul. Comput. 50 (8), 2213-2231, 2021.
4. [4] D. E. Farrar and R. R. Glauber, Multicollinearity in regression analysis: the problem revisited. Rev. Econ. Stat. 49 (1), 92-107, 1967.
5. [5] D. A. Belsley, E. Kuh and R. E. Welsch, Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. John Wiley & Sons, New York, 2005.
6. [6] R. F. Gunst, Regression analysis with multicollinear predictor variables: definition, detection, and effects. Commun. Stat. Theory Methods 12 (19), 2217-2260, 1983.
7. [7] A. E. Hoerl and R. W. Kennard, Ridge regression: applications to nonorthogonal problems. Technometrics 12 (1), 69-82, 1970.
8. [8] S. E. Wasti, Pakistan Economic Survey 20172018. Finance Division, Government of Pakistan, Islamabad, 2017.

9. [9] R. Frisch, Wirtschaftsrechnung und Input-Output-Analyse. G. Fisher Verlag, Jena, 1934.
10. [10] R. D. Cook, Detection of influential observation in linear regression. Technometrics 19 (1), 15-18, 1977.
11. [11] D. N. Gujarati and D. C. Porter, Basic Econometrics. McGraw-Hill, New York, 2009.
12. [12] R. F. Gunst and J. T. Webster, Regression analysis and problems of multicollinearity. Commun. Stat. Theory Methods 4 (3), 277-292, 1975.
13. [13] B. M. G. Kibria, Performance of some new ridge regression estimators. Commun. Stat. Simul. Comput. 32 (2), 419-435, 2003.
14. [14] G. Khalaf and G. Shukur, Choosing ridge parameter for regression problems. Commun. Stat. Simul. Comput. 34 (3), 627-636, 2005.
15. [15] M. A. Alkhamisi and G. Shukur, Developing ridge parameters for SUR model. Commun. Stat. Theory Methods 37 (4), 544-564, 2008.
16. [16] A. K. M. E. Saleh, B. M. G. Kibria and F. George, Comparative study of lasso, ridge regression, preliminary test and Stein-type estimators for the sparse Gaussian regression model. Stat. Optim. Inf. Comput. 7 (4), 626-641, 2019.
17. [17] I. S. Dar and S. Chand, New heteroscedasticity-adjusted ridge estimators in linear regression model. Commun. Stat. Theory Methods 52 (0), 1-15, 2023.
18. [18] J. S. Long and L. H. Ervin, Using heteroscedasticity consistent standard errors in the linear regression model. Am. Stat. 54 (3), 217-224, 2000.
19. [19] W. H. Greene and D. A. Hensher, A latent class model for discrete choice analysis: contrasts with mixed logit. Transp. Res. Part B Methodol. 37 (8), 681-698, 2003.
20. [20] G. Trenkler, A curious result on least squares estimation in linear regression. Int. J. Math. Educ. Sci. Technol. 20 (3), 475-489, 1989.
21. [21] M. I. Alheety and B. M. G. Kibria, On the Liu and almost unbiased Liu estimators in the presence of multicollinearity with heteroscedastic or correlated errors. Surv. Math. Appl. 4, 155-167, 2009.
22. [22] M. A. Alkhamisi, Ridge estimation in linear models with heteroskedastic errors. Sankhya B 74 (2), 302-314, 2012.
23. [23] A. V. Dorugade, Adjusted ridge estimator and comparison with Kibria’s method in linear regression. J. Assoc. Arab Univ. Basic Appl. Sci. 21, 96-102, 2016.
24. [24] V. Sevinç and A. Göktaş, A comparison of different ridge parameters under both multicollinearity and heteroscedasticity. Suleyman Demirel Univ. Fen Bilim. Enstit. Derg. 23 (2), 381-389, 2019.
25. [25] M. J. Silvapulle, Robust ridge regression based on an M-estimator. Aust. J. Stat. 33 (3), 319-333, 1991.
26. [26] H. Ertaş, A modified ridge M-estimator for linear regression model with multicollinearity and outliers. Commun. Stat. Simul. Comput. 47 (4), 1240-1250, 2018.
27. [27] M. Suhail, S. Chand and B. M. G. Kibria, Quantile-based robust ridge M-estimator for linear regression model in presence of multicollinearity and outliers. Commun. Stat. Simul. Comput. 50 (11), 3194-3206, 2021.
28. [28] D. Wasim, S. A. Khan, M. Suhail and M. Shabbir, New penalized M-estimators in robust ridge regression: real life applications using sports and tobacco data. Commun. Stat. Simul. Comput. 54 (5), 1-20, 2023.
29. [29] M. Suhail, S. Chand and M. Aslam, New quantile based ridge M-estimator for linear regression models with multicollinearity and outliers. Commun. Stat. Simul. Comput. 52 (4), 1417-1434, 2023.
30. [30] D. Wasim, S. A. Khan and M. Suhail, Modified robust ridge M-estimators for linear regression models: an application to tobacco data. J. Stat. Comput. Simul. 93 (15), 2703-2724, 2023.
31. [31] A. Majid, M. Amin, M. Aslam and S. Ahmad, New robust ridge estimators for the linear regression model with outliers. Commun. Stat. Simul. Comput. 52 (10), 4717- 4738, 2023.
32. [32] M. Aslam, Performance of Kibria’s method for the heteroscedastic ridge regression model: some Monte Carlo evidence. Commun. Stat. Simul. Comput. 43 (4), 673-686, 2014.
33. [33] M. Aslam, Using heteroscedasticity-consistent standard errors for the linear regression model with correlated regressors. Commun. Stat. Simul. Comput. 43 (10), 2353-2373, 2014.
34. [34] W. H. Greene, Econometric Analysis. Pearson Education India, New Delhi, 2003.
35. [35] A. E. Hoerl, R. W. Kennard and K. F. Baldwin, Ridge regression: some simulations. Commun. Stat. Theory Methods 4 (2), 105-123, 1975.
36. [36] J. F. Lawless and P. Wang, A simulation study of ridge and other regression estimators. Commun. Stat. Theory Methods 5 (4), 307-323, 1976.
37. [37] R. R. Hocking, F. M. Speed and M. J. Lynn, A class of biased estimators in linear regression. Technometrics 18 (4), 425-437, 1976.
38. [38] G. Khalaf, K. Månsson and G. Shukur, Modified ridge regression estimators. Commun. Stat. Theory Methods 42 (8), 1476-1487, 2013.
39. [39] D. C. Montgomery, E. A. Peck and G. G. Vining, Introduction to Linear Regression Analysis. John Wiley & Sons, Hoboken, 2021.
40. [40] P. J. Huber and E. M. Ronchetti, Robust Statistics. John Wiley & Sons, Hoboken, 2011.
41. [41] G. C. McDonald and D. I. Galarneau, A Monte Carlo evaluation of some ridge-type estimators. J. Am. Stat. Assoc. 70 (350), 407-416, 1975.
42. [42] D. G. Gibbons, A simulation study of some ridge estimators. J. Am. Stat. Assoc. 76 (373), 131-139, 1981.
43. [43] S. Perez-Melo and B. M. G. Kibria, On some test statistics for testing the regression coefficients in presence of multicollinearity: a simulation study. Stats 3 (1), 40-55, 2020.
44. [44] T. S. Breusch and A. R. Pagan, A simple test for heteroscedasticity and random coefficient variation. Econometrica 47 (5), 1287-1294, 1979.
45. [45] B. Gültay and S. Kaçranlar, Mean square error comparisons of the alternative estimators for the distributed lag models. Hacet. J. Math. Stat. 44 (5), 1215-1233, 2015.
46. [46] J. P. Newhouse and S. D. Oman, An Evaluation of Ridge Estimators: A Report Prepared for United States Air Force Project Rand. RAND Corp., Santa Monica, 1971.