Reviving some geometric aspects of shrinkage estimation in linear models

Abstract

It is well known that the least squares estimator is the best linear unbiased estimator of the parameter vector in a classical linear model. But, it is `too long' as a vector and unreliable, confidence intervals are broad for some components especially in the case of multicollinearity. Shrinkage (contraction) type estimators are efficient remedial tools in order to solve problems caused by multicollinearity. In this study, we consider a class of componentwise shrunken estimators with typical members: Mayer and Willke's contraction estimator, Marquardt's principal component estimator, Hoerl and Kennard's ridge estimator, Liu's linear unified estimator and a discrete shrunken estimator. All estimators considered are "shorter" than the least squares estimator with respect to the Euclidean norm, biased, but insensitive to multicollinearity and admissible within the set of linear estimators with respect to unweighted squared error risk. Some behaviors of these estimators are illustrated geometrically by tracing their trajectories as functions of shrinkage factors in a two- dimensional parameter space.

Keywords

• Contraction estimator,Liu estimator,principal component estimator,Ridge regression

References

1. Akdeniz F. and Kaçıranlar, S., On the almost unbiased generalized Liu estimator and unbiased estimation of the Bias and MSE, Communications in Statistics- Theory and Methods, (1995) 24(7): 1789-1797.
2. Alheety, M.I. and Kibria, B.M.G., Modified Liu-Type Estimator Based on (r-k) Class Estimator, Communications in Statistics---Theory and Methods, (2013) 42: 304--319.
3. Gross, J. Linear Regression, Springer, .2003.
4. Hoerl, A. E., Optimum solution to many variables equations. Application of ridge analysis to regression problems, Chemical Engineering Progress, (1959) 55, 69-78.
5. Hoerl, A. E., Application of ridge analysis to regression problems, Chemical Engineering Progress, (1962) 58, 54-59.
6. Hoerl, A. E. and Kennard, R. W., Ridge regression: biased estimation for nonorthogonal problems. Technometrics (1970) 12, 55-67.
7. Hoerl, A. E. and Kennard, R. W., Ridge regression: applications to nonorthogonal problems, Technometrics (1970) 12, 69-82.
8. Hoerl, A. E. Kennard, R. W. and Baldwin, F. K., Ridge regression: Some simulations, Communications in Statistics--Theory and Methods, (1975) 4, 105-123.

9. Hoerl, A. E. and Kennard, R. W., Ridge regression: Iterative estimation of the biasing parameter, Communications in Statistics--Theory and Methods, (1976) A5(1), 77-78.
10. Hoerl, A. E. and Kennard, R. W., Ridge regression-1979. Mimeo document -Bibliography, (1979) 12-77.
11. Hoerl, A. E. and Kennard, R.W., Ridge regression-1980. Advances, algorithms and applications, American Journal of Mathematical and Management Sciences, (1981) 1(1), 5-83.
12. Hoerl, A. E., Ridge Analysis 25 Years Later, The American Statistician, (1985) 39(3),186-192.
13. James, W. and Stein, C., Estimation with quadratic loss, Proceedings of the Fourth Berkeley Symposium, University of California Press, (1961) 361-379.
14. Liu, K. J., A new class of biased estimate in linear regression, Communications in Statistics--Theory and Methods, (1993), 22, 393-402.
15. Liu, K.J., Using Liu type estimator to combat multicollinearity, Communications in Statistics--Theory and Methods, (2003) 32(5), 1009-1020.
16. Marquardt, D.W., Generalized inverses, ridge regression, biased linear estimation and nonlinear estimation, Technometrics (1970) 12, 591-612.
17. Marquardt, D.W. and Snee, R.D., Ridge regression in practice. The American Statistician, (1975) 29, 3-20.
18. Meyer, L.S. and Wilke, T.A., On biased estimation in linear models, Technometrics (1973) 15, 497-508.
19. Öztürk, F., A discrete shrinking method as alternative to least squares, Communications de la Faculté des Sciences de l'University d'Ankara, (1984), 33, 179-185.
20. Öztürk, F. and Akdeniz, F., Ill-conditioning and multicollinearity, Linear Algebra and its Applications, (2000) 321, 295-305.
21. Rao, C.R., Estimation of parameters in a linear model, The Annals of Statistics, (1976) 4, 1023-1037.
22. Swindel, B.F., Instability of regression coefficients illustrted , The American Statistician, (1974) 28(2), 63-65.
23. Swindel, B.F., Good ridge estimators based on prior informatin, Communications in Statistics--Theory and Methods, (1976) A5(11), 1065-1075.
24. Swindel, B.F., Geometry of ridge regression illustrated, The American Statistician, (1981) 35(1), 12-15.
25. Theobald, CM, Generalization of Mean Square Error Applied to Ridge Regression, Journal of the Royal Statistical Society Series B, (1974) 36(1), 103-106.
26. Trenkler, G., An iteration estimator for the linear model, Computational Statistics, Physica Verlag, (1978) 125-131.