PERFORMANCE OF FACTOR RETENTION METHODS IN SKEWED DISTRIBUTIONS

Abstract

This research aims to evaluate the performance of dimensionality determination methods under various simulation conditions. Therefore, dimensionality determination methods were compared, including optimal parallel analysis, MAP, HULL, EGA (TMFG) estimation, EGA (glasso) estimation, and comparison data forest method. The type of distribution, sample size, number of items per factor, number of categories, and measurement model were specified as simulation conditions in the study. For each condition, 100 replications were conducted. A fully crossed simulation design was employed in the study. The results of this study, which examined the performance of factor determination methods under skewed distributions, indicated that the HULL method had the highest average considering the average accuracy values of all conditions. Meanwhile, the HULL method had the lowest RB average. However, no method demonstrated adequate performance under all conditions. This study examined one-factor and two-factor structures with interfactor correlations of 0.00 and 0.30. Considering structures with more than two factors in education and psychology, future research could focus on working with data exhibiting skewed distributions involving more factors and items to compare the performance of methods.

Keywords

• factor retention
• MAP
• HULL
• comparison factor forest
• EGA

ÇARPIK DAĞILIMLARDA FAKTÖR SAYISI BELİRLEME YÖNTEMLERİNİN PERFORMANSLARININ İNCELENMESİ

Abstract

Bu araştırmanın amacı faktör sayısı belirleme yöntemlerinin çeşitli simülasyon koşulları altında performanslarını değerlendirmektir. Bu amaç doğrultusunda boyutluluk belirleme yöntemlerinden optimal paralel analiz, MAP, HULL, EGA (TMFG) kestirimi, EGA (Glasso) kestirimi ve comparison data forest yöntemi karşılaştırılmıştır. Çalışmada simülasyon koşulları olarak dağılımın türü, örneklem büyüklüğü, faktör başına düşen madde sayısı, kategori sayısı ve ölçme modeli belirlenmiştir. Çalışmada her bir koşul için 100 replikasyon yapılmıştır. Çalışmada tamamen çaprazlanmış simülasyon deseni kullanılmıştır. Çarpık dağılımlarda faktör sayısı belirleme yöntemlerinin performanslarının incelendiği bu çalışma sonucunda tüm koşulların accuracy değerlerinin ortalaması dikkate alındığında en yüksek ortalamaya HULL yönteminin sahip olduğu görülmüştür. Aynı zamanda en düşük RB ortalaması da HULL yöntemindedir. Ancak tüm koşullarda yeterli performansı gösteren bir yöntemin olmadığı söylenebilir. Diğer bir deyişle her koşulda doğru sonucu verecek bir yöntem bulunmamaktadır. Bu çalışmada tek faktörlü, faktörler arası korelasyonu 0.00 ve 0.30 olan iki faktörlü yapılar incelenmiştir. Eğitimde ve psikolojide ikiden fazla faktör sayısına sahip yapılar göz önünde bulundurulduğunda gelecekteki araştırmalarda çarpık dağılım gösteren verilerde daha fazla faktör ve madde sayısıyla çalışılarak yöntemlerin performansları karşılaştırılabilir.

Keywords

• Faktör sayısı belirleme
• MAP
• HULL
• karşılaştırmalı faktör forest
• EGA

References

1. Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13(2), 186–203. https://doi.org/10.1207/s15328007sem1302_2
2. Cho, S.-J., Li, F., & Bandalos, D. L. (2009). Accuracy of the parallel analysis procedure with polychoric correlations. Educational and Psychological Measurement, 69(5), 748–759. https://doi.org/10.1177/0013164409332229
3. Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10(7), 27–29. https://doi.org/10.7275/jyj1-4868
4. Curran, P. J., West, S. G., & Finch, J. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16–29. https://doi.org/10.1037/1082-989X.1.1.16
5. Finch, W. H. (2020). Using Fit Statistic Differences to Determine the Optimal Number of Factors to Retain in an Exploratory Factor Analysis. Educational and Psychological Measurement, 80(2), 217–241. https://doi.org/10.1177/0013164419865769.
6. Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466–491. https://doi.org/10.1037/1082-989X.9.4.466
7. Foldnes, N., & Grønneberg, S. (2017). The asymptotic covariance matrix and its use in simulation studies. Structural Equation Modeling: A Multidisciplinary Journal, 24(6), 881–896. https://doi.org/10.1080/10705511.2017.1341320
8. Forero, C. G., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A monte carlo study comparing DWLS and ULS estimation. Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 625–641. https://doi.org/10.1080/10705510903203573

9. Garrido, L. E., Abad, F. J., & Ponsoda, V. (2011). Performance of Velicer’s minimum average partial factor retention method with categorical variables. Educational and Psychological Measurement, 71(3), 551–570. https://doi.org/10.1177/0013164410389489
10. Golino, H. F., & Christensen, A. P. (2020). EGAnet: Exploratory Graph Analysis – A framework for estimating the number of dimensions in multivariate data using network psychometrics.
11. Golino, H. F., & Epskamp, S. (2017). Exploratory graph analysis: A new approach for estimating the number of dimensions in psychological research. PLOS ONE, 12(6), 1–26. https://doi.org/10.1371/journal.pone.0174035.
12. Goretzko, D., & Bühner, M. (2020). One model to rule them all? Using machine learning algorithms to determine the number of factors in exploratory factor analysis. Psychological Methods, 25(6), 776–786.
13. Goretzko, D., & Bühner, M. (2022). Factor retention using machine learning with ordinal data. Applied Psychological Measurement, 46(5), 406–421. https://doi.org/10.1177/01466216221089345
14. Goretzko, D., Pham, T. T. H., & Bühner, M. (2021). Exploratory factor analysis: Current use, methodological developments and recommendations for good practice. Current Psychology, 40(7), 3510–3521. https://doi.org/10.1007/s12144-019-00300-2
15. Goretzko, D., & Ruscio, J. (2023). The comparison data forest: A new comparison data approach to determine the number of factors in exploratory factor analysis. Behavior Research Methods. https://doi.org/10.3758/s13428-023-02122-4
16. Gorsuch, R. L. (1974). Factor analysis. W. B. Saunders.
17. Haslbeck, J. M. B., & van Bork, R. (2022). Estimating the number of factors in exploratory factor analysis via out-of-sample prediction errors. Psychological methods, 10.1037/met0000528. Advance online publication. https://doi.org/10.1037/met0000528.
18. Henson, R., & Roberts, J. (2006). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Educational and Psychological Measurement, 66(3), 393–416.
19. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179–185. https://doi.org/10.1007/BF02289447
20. Kılıç, A. F., & Uysal, İ. (2019). Comparison of factor retention methods on binary data: A simulation study. Turkish Journal of Education, 8(3), 160–179. https://doi.org/10.19128/turje.518636
21. Koyuncu, İ., & Kılıç, A. F. (2019). The use of exploratory and confirmatory factor analyses: A document analysis. Eğitim ve Bilim, 44(198), 361–388. https://doi.org/10.15390/EB.2019.7665
22. Kılıç, A. F., & Uysal, İ. (2021). Faktör çıkarma yöntemlerinin paralel analiz sonuçlarına etkisi. Trakya Eğitim Dergisi, 11(2), 926–942. https://doi.org/10.24315/tred.747075.
23. Montoya, A. K., & Edwards, M. C. (2021). The poor fit of model fit for selecting number of factors in exploratory factor analysis for scale evaluation. Educational and Psychological Measurement, 81(3), 413–440. https://doi.org/10.1177/0013164420942899.
24. Ledesma, R. D., Valero-Mora, P., & Macbeth, G. (2015). The Scree Test and the Number of Factors: a Dynamic Graphics Approach. The Spanish journal of psychology, 18, E11. https://doi.org/10.1017/sjp.2015.13.
25. Lee, C., Park Y., & Cho B. (2023). Use of exploratory graph analysis in inspecting the dimensionality of the revised University of California Los Angeles (R-UCLA) Loneliness Scale Among Older Adults. Res Gerontol Nurs.16(1):15-20. https://doi.org/10.3928/19404921-20230104-03.
26. Li, C.-H. (2016). The performance of ML, DWLS, and ULS estimation with robust corrections in structural equation models with ordinal variables. Psychological Methods, 21(3), 369–387. https://doi.org/10.1037/met0000093
27. Li, Y., Wen, Z., Hau, K.-T., Yuan, K.-H., & Peng, Y. (2020). Effects of cross-loadings on determining the number of factors to retain. Structural Equation Modeling: A Multidisciplinary Journal, 27(6), 841–863. https://doi.org/10.1080/10705511.2020.1745075
28. Lorenzo-Seva, U., Timmerman, M. E., & Kiers, H. A. L. (2011). The Hull method for selecting the number of common factors. Multivariate Behavioral Research, 46(2), 340–364. https://doi.org/10.1080/00273171.2011.56452
29. Lozano, L. M., García-Cueto, E., & Muñiz, J. (2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology, 4(2), 73–79. https://doi.org/10.1027/1614-2241.4.2.73
30. Montoya, A. K., & Edwards, M. C. (2021). The Poor Fit of Model Fit for Selecting Number of Factors in Exploratory Factor Analysis for Scale Evaluation. Educational and Psychological Measurement, 81(3), 413–440. https://doi.org/10.1177/0013164420942899
31. Muthén, B. O., & Kaplan, D. (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38(2), 171–189. https://doi.org/10.1111/j.2044-8317.1985.tb00832.x
32. Navarro-Gonzalez, D., & Lorenzo-Seva, U. (2021). EFA.MRFA: Dimensionality assessment using minimum rank factor analysis. https://cran.r-project.org/package=EFA.MRFA
33. O’Connor, B. P. (2022). EFA.dimensions: Exploratory factor analysis functions for assessing dimensionality. https://cran.r-project.org/package=EFA.dimensions
34. R Core Team. (2022). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. https://www.r-project.org
35. Reio, T. G., & Shuck, B. (2015). Exploratory Factor Analysis: Implications for Theory, Research, and Practice. Advances in Developing Human Resources, 17(1), 12–25. https://doi.org/10.1177/1523422314559804
36. Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
37. Ruscio, J., & Roche, B. (2012). Determining the number of factors to retain in an exploratory factor analysis using comparison data of known factorial structure. Psychological Assessment, 24(2), 282–292. https://doi.org/10.1037/a0025697
38. Schmitt, T. A., & Sass, D. A. (2011). Rotation Criteria and Hypothesis Testing for Exploratory Factor Analysis: Implications for Factor Pattern Loadings and Interfactor Correlations. Educational and Psychological Measurement, 71(1), 95–113. https://doi.org/10.1177/0013164410387348
39. Svetina, D. (2011). Assessing dimensionality in complex data structures: A performance comparison of DETECT and NOHARM procedures. Yayınlanmamış doktora tezi, Arizona State Üniversitesi, Arizona.
40. Tabachnick, B. G., & Fidell, L. S. (2019). Using multivariate statistics (7th ed.). Pearson.
41. Timmerman, M. E., & Lorenzo-Seva, U. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16(2), 209–220. https://doi.org/10.1037/a0023353
42. West, S. G., Finch, J. F., & Curran, P. J. (1995). Structural equation models with non-normal variables: Problems and remedies. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications. Sage.
43. Yang, Y., & Xia, Y. (2015). On the number of factors to retain in exploratory factor analysis for ordered categorical data. Behavior Research Methods, 47(3), 756–772. https://doi.org/10.3758/s13428-014-0499-2