A Comparative Study of Penalized and Classical Variable Selection Methods

Abstract

In linear regression models, multicollinearity affects regression parameter estimates and can lead to misleading results in selecting the true model. This study, therefore, undertakes a comparison of the performance of various variable selection methods under multicollinearity. This simulation study evaluates the performance of classical criteria (Cp, AIC, AICC, ICOMP) and penalized regression techniques (LASSO, Ridge, Elastic Net, SCAD), as well as Partial Least Squares Regression (PLSR) for variable selection. The primary objective is to assess each method's effectiveness in accurately identifying the true model across varying levels of variance both in the presence and absence of multicollinearity. The results indicate that, in scenarios of multicollinearity, penalized methods and PLSR tend to yield results similar to those obtained in the absence of multicollinearity in the case of low variances, where they perform well in accurately estimating the true coefficients. In the presence of multicollinearity, Ridge regression emerges as the most successful method at a variance level of σ²=25. Moreover, ICOMP consistently outperforms classical selection criteria in settings of multicollinearity and high variance, while criteria such as Cp, AIC, and AICC exhibit diminished performance in this case.

Keywords

• Classical variable selection methods
• Model selection criterion
• Penalized regression methods
• PLSR

References

1. Mallows, C.L., “Some comments on Cp”, Technometrics, 15(4):661–675, (1973). DOI: https://doi.org/10.2307/1267380
2. Gilmour, S.G., “The interpretation of mallow’s Cp-statistic”, Journal of the Royal Statistical Society. Series D (The Statistician), 45(1):49-56, (1996). DOI: https://doi.org/10.2307/2348411
3. Akaike, H., “A new look at the statistical model identification”, IEEE Transactions on Automatic Control, 19:716-723, (1974). DOI: http://dx.doi.org/10.1109/TAC.1974.1100705
4. Çetin, M.C., and Erar, A., “Variable selection with akaike information criteria: a comparative study”, Hacettepe Journal of Mathematics and Statistics, 31:89-97, (2002). https://izlik.org/JA53KY84BB
5. Hurvich, C.M., and Tsai, C.L., “Regression and time series model selection in small samples”, Biometrika, 76(2):297-307, (1989). DOI: https://doi.org/10.2307/2336663
6. Sugiura, N., “Further analysis of the data by akaike's information criterion and the finite corrections”, Communications in Statistics—Theory and Methods A, 7:13-26, (1978). DOI: https://doi.org/10.1080/03610927808827599
7. Aşıkgil B., “Information complexity criterion for order determination in autoregressive models”, Hacettepe Journal of Mathematics and Statistics, 40(4):601-607, (2011). https://izlik.org/JA42NA98JE
8. Bozdogan, H., “ICOMP: a new model selection criterion”, In: Bock H. (Ed.), Classification and Related Methods of Data Analysis, North-Holland: Amsterdam, 599-608, (1988).

9. Bozdogan, H., “Akaike's information criterion and recent developments in information complexity”, Journal of Mathematical Psychology, 44(1):62-91, (2000). DOI: https://doi.org/10.1006/jmps.1999.1277
10. Bozdogan, H., “Intelligent statistical data mining with information complexity and genetic algorithms”, In: Bozdogan H. (Ed.), Statistical Data Mining and Knowledge Discovery, FL: Chapman and Hall/CRC, Boca Raton, Florida, 15–56, (2004).
11. Hoerl, A.E., and Kennard, R.W., “Ridge regression: biased estimation for nonorthogonal problems”, Technometrics, 12(1):55–67, (1970). DOI: https://doi.org/10.2307/1267351
12. Nayem, H.M., Aziz, S., and Golam Kibria, B.M., “Comparison among ordinary least squares, ridge, lasso, and elastic net estimators in the presence of outliers: simulation and application”, International Journal of Statistical Sciences, 24(2):25-48, (2024). DOI: https://doi.org/10.3329/ijss.v24i20.78212
13. Tibshirani, R., “Regression shrinkage and selection via the lasso”, Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267–288, (1996). DOI: https://doi.org/10.1111/j.2517-6161.1996.tb02080.x
14. Fu, W., and Knight, K., “Asymptotics for lasso-type estimators”, The Annals of Statistics, 28(5):1356–1378, (2000). DOI: https://doi.org/10.1214/aos/1015957397
15. Yuan, M., and Lin, Y., “On the non-negative garrotte estimator”, Journal of the Royal Statistical Society. Series B: Statistical Methodology, 69(2):143–161, (2007). DOI: https://doi.org/10.1111/j.1467-9868.2007.00581.x
16. Lounici, K., “Sup-norm convergence rate and sign concentration property of lasso and dantzig estimators”, Electronic Journal of Statistics, 2:90-102, (2008). DOI: https://doi.org/10.1214/08-EJS177
17. Zou, H., and Hastie, T., “Regularization and variable selection via the elastic net”, Journal of the Royal Statistical Society. Series B: Statistical Methodology, 67(2):301–320, (2005). DOI: https://doi.org/10.1111/j.1467-9868.2005.00503.x
18. Park, H., and Konishi, S., “Robust logistic regression modelling via the elastic net-type regularization and tuning parameter selection”, Journal of Statistical Computation and Simulation, 86(7):1450-1461, (2015). DOI: https://doi.org/10.1080/00949655.2015.1073290
19. Liu, W., and Li, Q., “An efficient elastic net with regression coefficients method for variable selection of spectrum data”, PLOS ONE, 12(2):1-13, (2017). DOI: https://doi.org/10.1371/journal.pone.0171122
20. Emmert-Streib, F., and Dehmer, M., “Understanding statistical hypothesis testing: the logic of statistical inference”, Machine Learning and Knowledge Extraction, 1: 945–961, (2019). DOI: https://doi.org/10.3390/make1030054
21. Fan, J., and Li, R., “Variable selection via nonconcave penalized likelihood and its oracle properties”, Journal of the American Statistical Association, 96(456):1348–1360, (2001). DOI: https://doi.org/10.1198/016214501753382273
22. Li, T., Han F., Guo, J., and Wu, J., “Comparative analysis of regularization methods for predicting student certification”, In: Saqr M. and López-Pernas S. (Eds), Advanced Learning Analytics Methods: AI, Precision and Complexity. Springer. https://lamethods.org/book2/chapters/ch05-regularization/ch05-regularization.html, (2025).
23. Zhang, K., Yin, F., and Xiong, S., “Comparisons of penalized least squares methods by simulations”, (2014). DOI: https://doi.org/10.48550/arXiv.1405.1796
24. De Jong, S., “SIMPLS: An alternative approach to partial least squares regression”, Chemometrics and Intelligent Laboratory Systems, 18(3):251-263, (1993). DOI: https://doi.org/10.1016/0169-7439(93)85002-X
25. Mevik, B.H., and Wehrens, R., “The pls package: principal component and partial least squares regression in R”, Journal of Statistical Software, 18(2):1-23, (2007). DOI: https://doi.org/10.18637/jss.v018.i02
26. Polat, E., and Gunay, S., “The comparison of partial least squares regression and ridge regression with multiple linear regression for predicting pm10 concentration level based on meteorological parameters”, Journal of Data Science, 13(4):663–692, (2015). DOI: https://doi.org/10.6339/JDS.201510_13(4).0003
27. Montgomery, D.C., and Peck, E.A., “Introduction to Linear Regression Analysis”, John Wiley and Sons, Inc., New York, (1982).