SCAD-Ridge penalized likelihood estimators for ultra-high dimensional models

Abstract

Extraction of as much information as possible from huge data is a burning issue in the modern statistics due to more variables as compared to observations therefore penalization has been employed to resolve that kind of issues. Many achievements have already been made by such penalization techniques. Due to the large number of variables in many research areas declare it a high dimensional problem and with this the sample correlation becomes very large. In this paper, we studied the maximum likelihood estimation of variable selection under smoothly clipped absolute deviation (SCAD) and Ridge penalties with ultra-high dimension settings to solve this problem. We established the oracle property of the proposed model under some conditions by following the theoretical method of Kown and Kim (2012) [19]. These result can greatly broaden the application scope of high-dimension data. Numerical studies are discussed to assess the performance of the proposed method. The SCAD-Ridge given better results than the Lasso, Enet and SCAD.

Keywords

• Maximum likelihood estimation,Oracle property,SCAD-Ridge penalization,Ultra-high dimension,Variable selection

References

1. Amin, M., Song L, Milton A.T, Xiaoguang W. Combined penalized quantile regression in high dimensional models. Pakistan Journal Statistics. 31, 4970, 2015.
2. Antoniadis, A. Wavelets in Statistics: A Review (with discussion) . Journal of the Italian Statistical Association,6, 97144, 1997.
3. Breheny, P. and Huang, J. Coordinate descent algorithms for nonconvex penalized regression with application to biological feature selection. Ann. Appl. Stat. 5, 232253, 2011.
4. Dong, Y., Song L. X., Wang, M. Q. and Xu, Y. Combined-penalized likelihood estimations with a diverging number of parameters. Journal of Applied Statistics. 41, 12741285, 2014.
5. Donoho, D. L. and Johnstone, I. M. Ideal spatial adaptation via wavelet shrinkage. Biometrika 81, 425455, 1994.
6. Efron, B., Hasti, T. and Johnstone, I. Least angle regression. The Annals of Statistics 32, 407499, 2004.
7. Hoerl, A. E. and Kennard, R. W. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12, 5567, 1970.
8. Fan, J. Comments on 'Wavelets in Statistics: A Review' by A. Antoniadis. Journal of the Italian Statistical Association 6, 131138, 1997.

9. Fan, J. and Li, R. Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96, 13481360, 2001.
10. Fan, J. and Peng, H. Nonconcave penalized likelihood with diverging number of parameters. The Annals of Statistics 32, 928961, 2004.
11. Fan, J. and Lv, J. Sure independence screening for ultra-high dimensional feature space. J. Roy. Statist. Soc. Ser. B. 70, 849911, 2008.
12. Fan, J. and Lv, J. A selective overview of variable selection in high dimensional feature space (invited review article). Statistica Sinica 20, 101148, 2010.
13. Friedman, J., Hastie, T., Hoing, H. and Tibshirani, R. Pathwise coordinate optimization. Ann. Appl. Stat. 1, 302332, 2007.
14. Friedman, J., Hastie, T. and Tibshirani, R. Regularization paths for generalized linear models via coordinate descent. J. Statist. Softw. 33, 122, 2010.
15. Fu, W. J. Penalized regressions: The Bridge versus the Lasso. Journal of Computational and Graphical Statistics 7, 397416, 1998.
16. Harrison, D., Rubinfeld, D. L. Hedonic prices and the demand for clean air. J. Environ. Economics and Management. 51, 81102, 1978.
17. Huang, J., Breheny, P., Ma, S. G. and Zhang, C. H. The Mnet Method for Variable Selection. The University of Iowa Department of Statistics and Actuarial Science Technical Report No. 4021, 2010.
18. Kim, Y., Choi, H. and Oh, H. Smoothly clipped absolute deviation on high dimensions. J. Amer. Statist. Assoc.103, 16561673, 2008.
19. Kwon, S., Kim, Y. D. (2012). Large Sample Properties of The SCAD-Penalized Maximum Likelihood Estimation on High Dimensiona. Statistica Sinica 22, 629653.
20. Meinshausen, N. and Buhlmann, P. High-dimensional graphs and variable selection with the Lasso. Ann. Statist.34, 14361462, 2006.
21. Rosset, S. and Zhu, J. Piecewise linear regularized solution paths. Ann. Statist. 35, 1012 1030, 2007.
22. Segerstedt, B. On ordinary ridge regression in generalized linear models. Communications in Statistics - Theory and Methods 21, 22272246, 1992.
23. Tibshirani, R. J. Regression shrinkage and selection via the Lasso. J. Roy. Statist. Soc. B 58, 267288, 1996.
24. Wang, X. M., Park, T. and Carriere, K. C. Variable selection via combined penalization for high-dimensional data analysis. Computational Statistics and Data Analysis 54, 22302243, 2010.
25. Zhao, P. and Yu, B. On model selection consistency of Lasso. J. Mach. Learn. Res. 7, 25412567, 2006.
26. Zou, H. and Hastie, T. Regularization and variable selection via the elastic net. J. Roy. Statist. Soc. B 67, 301320, 2005.
27. Zou, H. and Zhang, H. On the adaptive elastic-net with a diverging number of parameters. The Annals of Statistics 37, 17331751, 2009.