New approaches for choosing the ridge parameters

Abstract

Consider the standard multiple linear regression model $y=x\beta+\varepsilon.$ If the correlation matrix $x^tx$  is ill-conditioned, the ordinary least squared estimate (ols) $\hat{\beta}$ of $\beta$ is not the best choice. In this paper, multiple regularization  parameters for different coefficients in ridge regression are proposed. The Mean Squared Error (MSE) of a ridge estimate  based on the multiple regularization parameters is less than or equal to the MSE of the ridge estimate based on Hoerl and Kennard, 1970. The proposed approach, depending on
the condition numbers, leave's zero for the largest eigenvalue of $x^tx$ and gives the largest value  for the smallest eigenvalue of $x^tx.$
Furthermore, if $x^tx$ is nearly a unit matrix, $x^tx$  is not an ill-conditioned one. The proposed approach gives approximately the same results as the ols estimates. The proposed approach can also be modified to give other new ridge parameters. The modified approach depends on the eigenvalues of  $x^tx$ and  differ from the ridge parameter proposed by Khalaf and Shukur by a factor. The body fat data set has severe multicollinearity and is used to compare different approaches.

Keywords

• Ridge regression,multicollinearity,condition number,mean squared error,squared bias

References

1. Alkhamisi, M., Khalaf, G., Shukur, G., Some Modifications for Choosing Ridge Parameters, Communications in Statistics - Theory and Methods, 35 (11), 2005-2020, 2006.
2. Da Silva, J. L., Mexia, J. T., Ramos, L. P., On the Strong Consistency of Ridge Estimates, Communications in Statistics - Theory and Methods, 44 (3), 617-626, 2015.
3. Gisela, M., Kibria, B. M. G., On Some Ridge Regression Estimators: An Emirical Comparisons, Communications in Statistics - Simulation and Computation, 38 (3), 621--630, 2009.
4. Hocking, R. R., Methods and Applications of Linear Models, Wiley Series in Probability and Statistics, 2003.
5. Hoerl, A. E., Kennard, R. W., Ridge Regression: Biased Estimation for Nonorthogonal Problems, Technometrics, 12, 55-67, 1970.
6. Huang, C.-C. L., Jou, Y.-J., Cho, H.-J., VIF-based adaptive matrix perturbation method for heteroskedasticity-robust covariance estimators in the presence of multicollinearity, Communications in Statistics - Theory and Methods, http://dx.doi.org/10.1080/03610926.2015.1060340.
7. Khalaf, G., Mansson, K., Shukur, G., Modified Ridge Regression Estimators, Communications in Statistics - Theory and Methods, 42, 1476--1487, 2013.
8. Khalaf, G., Shukur, G., Choosing Ridge Parameter for Regression Problems, Communications in Statistics - Theory and Methods, 34 (5), 1177--1182, 2005.

9. Kibria, B. M. G., Performance of some new ridge regression estimators, Communications in Statistics - Theory and Methods, 32, 419-435, 2003.
10. Li, S., Sheng, Y., Li, Y., The quasi-stochastically constrained least squares method for ill linear regression, Communications in Statistics - Theory and Methods, 45 (2), 217-225, 2016.
11. Månsson, K., Shukur, G., Kibria, B. M. G., Performance of Some Ridge Regression Estimators for the Multinomial Logit Model, Communications in Statistics - Theory and Methods, http://dx.doi.org/10.1080/03610926.2013.784996.
12. NCSS Statistical Software, Chapter 335: Ridge Regression, NCSS, LLC. All Rights reserved.
13. Neter, J., Kutner, M. H., Nachtsheim, C., J., Wasserman, W., Applied Linear Statistical Models, 4th Edition, 1996.
14. Zeebari, Z., Shukur, G., On the Least Absolute Deviations Method for Ridge Estimation of SURE Models, Communications in Statistics - Theory and Methods, http://dx.doi.org/10.1080/03610926.2012.755203