Comparison of Least Squares and Principal Components Regression Analysis

Abstract

The aim of this study is to compare Least Squares Regression (LSR) and Principal Components Regression (PCR) results when multicollinearity is determined in a dataset.In order to examine the effect of the degree of multicollinearity in the study, 10 datasets with different levels of multicollinearity were derived. Each data set consists of three independent and one dependent variable, and the variables were derived from the standard normal distribution. The multicollinearity status in the derived data has been demonstrated by the commonly used metrics to determine multicollinearity. Least Squares and Principal Components Regression was applied to all datasets. When generating multicollinearity, all relationships were defined as positive in data simulation. However, the sign of the regression coefficients for the second (X2) and third (X3) independent variables were obtained as reverse (negative) as one of the expected effects of multicollinearity in Least Squares analysis. In the analysis of the Principal Components Regression, the sign of coefficients was found to be in the right direction (positive). The sign of the coefficients obtained from OLS and PCR were different and they also differed in magnitude. In addition, the standard errors of the coefficients in PCR results were lower than OLS results. The existence of multicollinearity must be examined while performing multiple linear regression analysis, and if multicollinearity is determined, one of the methods that can solve this situation should be used. Otherwise, the estimations to be made as a result of regression may lead to wrong results. In line with the results of this study, it is recommended to use Principal Components Regression instead of Least Squares regression in case of multicollinearity in the data.

Keywords

• Linear regression
• multicollinearity
• least squares
• principal components regression
• sample size.

References

1. [1] F. Akdeniz, Olasılık ve istatistik: Akademisyen Kitabevi, 2015.
2. [2] S. Şahinler, "En küçük kareler yöntemi ile doğrusal regresyon modeli oluşturmanın temel prensipleri," Mustafa Kemal Üniversitesi Ziraat Fakültesi Dergisi, vol. 5, pp. 57-73, 2000.
3. [3] D. C. Montgomery, E. A. Peck, and G. G. Vining, Introduction to linear regression analysis vol. 821: John Wiley & Sons, 2012.
4. [4] R. Özkale, "Çoklu İç İlişki ile İlgili Yöntemler," Doktora, Fen Bilimleri Enstitüsü, Çukurova Üniversitesi, 2007.
5. [5] I. Corp, "IBM SPSS statistics for windows, version 22.0," Armonk, NY: IBM Corp, 2013.
6. [6] R. Alpar, Spor, sağlık ve eğitim bilimlerinden örneklerle uygulamalı istatistik ve geçerlik-güvenirlik: Detay Yayıncılık, 2010.
7. [7] K. Özdamar, "SPSS ile Biyoistatistik. Kaan Kitabevi.. 4," Baskı. Eskişehir, pp. 315-368, 2001.
8. [8] J. Neter, M. H. Kutner, C. J. Nachtsheim, and W. Wasserman, Applied linear statistical models vol. 4: Irwin Chicago, 1996.

9. [9] S. Cankaya, "A comparative study of some estimation methods for parameters and effects of outliers in simple regression model for research on small ruminants," Tropical animal health and production, vol. 41, pp. 35-41, 2009.
10. [10] W. W. Hines, D. C. Montgomery, and D. M. G. C. M. Borror, Probability and statistics in engineering: John Wiley & Sons, 2008.
11. [11] R. Alpar, Uygulamalı çok değişkenli istatistiksel yöntemlere giriş-I: Bağırgan Yayımevi, 1997.
12. [12] M. Topal, E. Eyduran, A. M. Yağanoğlu, A. Sönmez, and S. Keskin, "Çoklu doğrusal bağlantı durumunda ridge ve temel bileşenler regresyon analiz yöntemlerinin kullanımı," Atatürk Üniversitesi Ziraat Fakültesi Dergisi, vol. 41, pp. 53-57, 2010.
13. [13] A. İşi, "Yanlı Tahmin Ediciler ve Kombinasyonları," Yüksek Lisans Tezi. Ankara: Gazi Üniversitesi Fen Bilimleri Enstitüsü, 2002.
14. [14] A. Göktaş and İ. Öznur, "Türkiye'de İşsizlik Oranının Temel Bileşenli Reegresyon Analizi ile Belirlenmesi " Sosyal Ekonomik Araştırmalar Dergisi, vol. 10, pp. 279-294, 2010.
15. [15] F. Aşkın, "Ortalama Artelyel Kan Basıncını Etkileyen Faktörlerin Temel Bileşenler Regresyonu İle Belirlenmesi," Yüksek Lisans, Fen Bilimleri Enstitüsü, Fırat Üniversitesi, 2011.
16. [16] N. Ortabaş, "Principal components in the problem of multicollineartity," DEÜ Fen Bilimleri Enstitüsü, 2001.
17. [17] A. Preece, D. Harborne, D. Braines, R. Tomsett, and S. Chakraborty, "Stakeholders in explainable AI," arXiv preprint arXiv:1810.00184, 2018.
18. [18] S. IBMCorp Ibm, "statistics for windows, version 25.0," Armonk, NY: IBM Corp, 2017.
19. [19] A. S. Albayrak, "Çoklu doğrusal bağlanti halİnde enküçük kareler teknİğİnİn alternatİfİ yanli tahmİn teknİklerİ ve bİr uygulama," Uluslararası Yönetim İktisat ve İşletme Dergisi, vol. 1, pp. 105-126, 2005.
20. [20] K. Orçanli, B. Bİrgören, and E. Oktay, "Çok Değişkenli Kalite Kontrolünde Süreç Tabanlı Temel Gösterimleri Yönteminin Hata Teriminde Kovaryansın Etkileri," Sosyal Bilimler Araştırma Dergisi, vol. 6, pp. 20-40.