Simulated Annealing Algorithm Based Ridge Estimator

Abstract

Multicollinearity is a significant problem in multiple linear regression. Different researchers have suggested biased estimators as a possible solution to address the issue of multicollinearity, and an example of a biased estimator is the ridge regression estimator. Estimating the bias parameter is an essential problem for the ridge regression estimator. This paper presents a new solution method that utilizes simulated annealing optimization to determine the optimal bias parameter as an alternative to the ridge regression bias value proposed by Hoerl and Kennard. We obtained the bias parameter estimation values using the proposed solution method, considering various dependency structures, sample sizes, variance, and number of variables.

Keywords

• Ridge regression
• Biasing parameter
• Simulated annealing
• Optimization
• Multicollinearity

References

1. [1] Frisch, R., “Statistical confluence analysis by means of complete regression system”, The Economic Journal, 45(180):741–742, (1935). DOI: https://doi.org/10.2307/2225583
2. [2] Montgomery, D.C., Peck, E.A., Vining, G.G., Introduction to Linear Regression Analysis, John Wiley & Sons, (2012).
3. [3] Khalaf, G., Iguernane, M., “Ridge regression and ill-conditioning”, Journal of Modern Applied Statistical Methods, 13(2):355–363, (2014). DOI: https://doi.org/10.22237/jmasm/1414815420
4. [4] Hoerl, A.E., Kennard, R.W., “Ridge regression: biased estimation for nonorthogonal problems”, Technometrics, 12(1):55–67, (1970). DOI: https://doi.org/10.2307/1267351
5. [5] Keijan, L., “A new class of biased estimate in linear regression”, Communications in Statistics - Theory and Methods, 22(2):393–402, (1993). DOI: https://doi.org/10.1080/03610929308831027
6. [6] Kibria, B.M.G., “Performance of some new ridge regression estimators”, Communications in Statistics Part B: Simulation and Computation, 32(2):419–435, (2003). DOI: https://doi.org/10.1081/SAC-120017499
7. [7] Khalaf, G., Shukur, G., “Choosing ridge parameter for regression problems”, Communications in Statistics - Theory and Methods, 34(5):1177–1182, (2006). DOI: https://doi.org/10.1081/STA-200056836
8. [8] Alkhamisi, A., Shukur, G., “A monte carlo study of recent ridge parameters”, Communications in Statistics - Simulation and Computation, 36(3):535–547, (2007). DOI: https://doi.org/10.1080/03610910701208619

9. [9] Dorugade, A.V., “New ridge parameters for ridge regression”, Journal of the Association of Arab Universities for Basic and Applied Sciences, 15(1):94–99, (2014). DOI: https://doi.org/10.1016/j.jaubas.2013.03.005
10. [10] Ahn, J.J., Byun, H.W., Oh, K.J., Kim, T.Y., “Using ridge regression with genetic algorithm to enhance real estate appraisal forecasting”, Expert Systems with Applications, 39(9):8369–8379, (2012). DOI: https://doi.org/10.1016/j.eswa.2012.01.183
11. [11] Kareem, Q. A., Algamal, Z. Y., “Generalized ridge estimator shrinkage estimation based on particle swarm optimization”, Iraqi Journal of Statistical Sciences, 17(2): 26–35, (2020). DOI: https://doi.org/10.33899/iqjoss.2020.167387
12. [12] Cagcag, O., Yolcu, U., Egrioglu, E., “A new robust regression method based on particle swarm optimization”, Communications in Statistics - Theory and Methods, 44(6):1270–1280, (2015). DOI: https://doi.org/10.1080/03610926.2012.718843
13. [13] Bas, E., Egrioglu, E., Uslu, V.R., “Shrinkage parameters for each explanatory variable found via particle swarm optimization in ridge regression”, Trends in Computer Science and Information Technology, 2(1):012–020, (2017). DOI: https://doi.org/10.17352/tcsit.000005
14. [14] Inan, D., Egrioglu, E., Sarica, B., Askin, O.E., Tez, M., “Particle swarm optimization based Liu-type estimator”, Communications in Statistics - Theory and Methods, 46(22):11358–11369, (2017). DOI: https://doi.org/10.1080/03610926.2016.1267759
15. [15] Inan, D., Sancar, N., “Particle swarm optimization based ridge logistic estimator”, Communications in Statistics - Simulation and Computation, 49(3):669–683, (2020). DOI: https://doi.org/10.1080/03610918.2020.1713361
16. [16] Kocasoy, G.İ., “Ridge ve Liu Regresyonda Tavlama Benzetimi Optimizasyonu Kullanılarak Yanlılık Parametrelerinin Elde Edilmesi ve Bazı Yanlılık Parametreleri ile Karşılaştırılması”, Master Thesis, Gazi University, Graduate School of Natural and Applied Sciences, Ankara, 11-14 (2023).
17. [17] Kennedy, J., Eberhart, R., “Particle swarm optimization”, in Proceedings of IEEE International Conference on Neural Networks, Perth, WA, Australia, (1995).
18. [18] Bozorg-Haddad, O., Solgi, M., Loaiciga, H.A., Meta-Heuristic and Evolutionary Algorithms for Engineering Optimization, John Wiley & Sons, (2017).
19. [19] Uslu, V.R., Egrioglu, E., Bas, E., “Finding optimal value for the shrinkage parameter in ridge regression via particle swarm optimization”, American Journal of Intelligent Systems, 4(4):142–147, (2014). DOI: https://doi.org/10.5923/j.ajis.20140404.03
20. [20] Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P., “Optimization by simulated annealing”, Science, 220(4598):671–680, (1983). DOI: https://doi.org/10.1126/science.220.4598.671
21. [21] Kumar, K., Zindani, D., Davim, J.P., Optimizing Engineering Problems through Heuristic Techniques, Taylor & Francis, (2020).
22. [22] Gad, A.G., “Particle swarm optimization algorithm and its applications: a systematic review”, Archives of Computational Methods in Engineering, 29(5):2531–2561, (2022). DOI: https://doi.org/10.1007/s11831-021-09694-4
23. [23] Hamami, M. G. M., Ismail, Z. H., “A systematic review on particle swarm optimization towards target search in the swarm robotics domain”, Archives of Computational Methods in Engineering, (2022). DOI: https://doi.org/10.1007/s11831-022-09819-3
24. [24] Ben-Ameur, W., “Computing the initial temperature of simulated annealing”, Computational Optimization and Applications, 29(3):369–385, (2003). DOI: https://doi.org/10.1023/B:COAP.0000044187.23143.bd
25. [25] Islam, M. S., Khatoon, M. T., Siddiquee, K. N., Yong, W. H., Huda, M. H., “Performance analysis of simulated annealing and genetic algorithm on systems of linear equations”, F1000Research, 10:1297, (2021). DOI: https://doi.org/10.12688/f1000research.73581.1
26. [26] Woods, H., Steinour, H. H., Starke, H. R., “Effect of composition of portland cement on heat evolved during hardening”, Industrial & Engineering Chemistry, 24(11):1207-1214, (1932). DOI: https://doi.org/10.1021/IE50275A002
27. [27] Longley, J.W., “An appraisal of least squares programs for the electronic computer from the point of view of the user”, Journal of the American Statistical Association, 62(319):819–841, (1967). DOI: https://doi.org/10.2307/2283673