Comparison of factor retention methods on binary data: A simulation study

Abstract

In this study, the purpose is to compare factor retention methods under simulation conditions. For this purpose, simulations conditions with a number of factors (1, 2 [simple]), sample sizes (250, 1.000, and 3.000), number of items (20, 30), average factor loading (0.50, 0.70), and correlation matrix (Pearson Product Moment [PPM] and Tetrachoric) were investigated. For each condition, 1.000 replications were conducted. Under the scope of this research, performances of the Parallel Analysis, Minimum Average Partial, DETECT, Optimal Coordinate, and Acceleration Factor methods were compared by means of the percentage of correct estimates, and mean difference values. The results of this study indicated that MAP analysis, as applied to both tetrachoric and PPM correlation matrices, demonstrated the best performance. PA showed a good performance with the PPM correlation matrix, however, in smaller samples, the performance of the tetrachoric correlation matrix decreased. The Acceleration Factor method proposed one factor for all simulation conditions. For unidimensional constructs, the  DETECT method was affected by both the sample size and average factor loading.

Keywords

• Exploratory factor analysis,Factor retention,Parallel analysis,Minimum average partial,DETECT

Faktör sayısını belirleme yöntemlerinin karşılaştırılması: Bir simülasyon çalışması

Abstract

Bu araştırmada faktör sayının belirlenmesi amacıyla geliştirilen yöntemlerin simülasyon koşulları altında karşılaştırılması amaçlanmıştır. Bu amaç için faktör sayısı (1, 2 [basit]), örneklem büyüklüğü (250, 1000 ve 3000), madde sayısı (20, 30), ortalama faktör yükü (0.50, 0.70) ve kullanılan korelasyon matrisi (Pearson Momentler Çarpımı [PPM] ve Tetrakorik) simülasyon koşulu olarak araştırılmıştır. Her bir koşul için 1000 replikasyon yapılmış ve üretilen 24000 veri seti için PPM ve tetrakorik korelasyon matrisi üzerinden analizler gerçekleştirilmiştir. Araştırma kapsamında Paralel Analiz, Kısmi Korelasyonların En Küçüğü, DETECT, Optimal Koordinat ve İvmelenme Faktörü yöntemlerinin performansları doğru kestirim yüzdesi ve ortalama fark değerleri üzerinden karşılaştırılmıştır. Araştırma sonucunda hem tetrakorik hem de PPM korelasyon matrisiyle yürütülen MAP analizi en iyi performansı göstermiştir. PA da PPM korelasyon matrisiyle iyi performans göstermiş ancak küçük örneklemde tetrakorik korelasyon matrisiyle performansı düşmüştür. DETECT yöntemi tek boyutlu yapılarda örneklem büyüklüğü ve ortalama faktör yükünden etkilenmiştir.

Keywords

• Açımlayıcı faktör analizi,Faktör sayısını belirleme,Paralel analiz,Kısmi korelasyonların en küçüğü,DETECT

References

1. Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13(2), 186–203. https://doi.org/10.1207/s15328007sem1302_2
2. Bennett, R. E., Rock, D. A., Braun, H. I., Frye, D., Spohrer, J. C., & Soloway, E. (1990). The relationship of expert-system scored constrained free-response items to multiple-choice and open-ended items. Applied Psychological Measurement, 14(2), 151–162. https://doi.org/10.1177/014662169001400204
3. Bennett, R. E., Rock, D. A., & Wang, M. (1991). Equivalence of free-response and multiple-choice items. Journal of Educational Measurement, 28(1), 77–92. https://doi.org/10.1111/j.1745-3984.1991.tb00345.x
4. Buja, A., & Eyuboglu, N. (1992). Remarks on parallel analysis. Multivariate Behavioral Research, 27(4), 509–540. https://doi.org/10.1207/s15327906mbr2704_2
5. Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1(2), 245–276. https://doi.org/10.1207/s15327906mbr0102_10
6. Cho, S.-J., Li, F., & Bandalos, D. L. (2009). Accuracy of the parallel analysis procedure with polychoric correlations. Educational and Psychological Measurement, 69(5), 748–759. https://doi.org/10.1177/0013164409332229
7. Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis (2nd ed.). London: Lawrence Erlbaum Associates.
8. Conway, J. M., & Huffcutt, A. I. (2003). A review and evaluation of exploratory factor analysis practices in organizational research. Organizational Research Methods, 6(2), 147–168. https://doi.org/10.1177/1094428103251541

9. Cota, A. A., Longman, R. S., Holden, R. R., Fekken, G. C., & Xinaris, S. (1993). Interpolating 95th percentile eigenvalues from random data: An empirical example. Educational and Psychological Measurement, 53(3), 585–596. https://doi.org/10.1177/0013164493053003001
10. DeMars, C. (2010). Item response theory. New York: Oxford University.
11. Dinno, A. (2014). Gently clarifying the application of Horn’s parallel analysis to principal component analysis versus factor analysis. Unpublished manuscript, School of Community Health, Portland State University, Oregon, USA. Retrieved from http://doyenne.com/Software/files/ PA_for_PCA_vs_FA.pdf
12. Fabrigar, L. R., & Wegener, D. T. (2012). Exploratory factor analysis. New York: Oxford University.
13. Feinberg, R. A., & Rubright, J. D. (2016). Conducting simulation studies in psychometrics. Educational Measurement: Issues and Practice, 35(2), 36–49. https://doi.org/10.1111/emip.12111
14. Finch, H., & Habing, B. (2005). Comparison of NOHARM and DETECT in item cluster recovery: Counting dimensions and allocating items. Journal of Educational Measurement, 42(2), 149–169. https://doi.org/10.1111/j.1745-3984.2005.00008
15. Finch, H., & Habing, B. (2007). Performance of DIMTEST- and NOHARM- based statistics for testing unidimensionality. Applied Psychological Measurement, 31(4), 292–307. https://doi.org/10.1177/0146621606294490
16. Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7(3), 286–299. https://doi.org/10.1037/1040-3590.7.3.286
17. Fraser, C., & McDonald, R. P. (1988). NOHARM: Least squares item factor analysis. Multivariate Behavioral Research, 23(2), 267–269. https://doi.org/10.1207/s15327906mbr2302_9
18. Garrido, L. E., Abad, F. J., & Ponsoda, V. (2011). Performance of Velicer’s minimum average partial factor retention method with categorical variables. Educational and Psychological Measurement, 71(3), 551–570. https://doi.org/10.1177/0013164410389489
19. Gilbert, N. (1999). Simulation: A new way of doing social science. American Behavioral Scientist, 42(10), 1485–1487. https://doi.org/10.1177/0002764299042010002
20. Glorfeld, L. W. (1995). An improvement on Horn’s parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement, 55(3), 377–393. https://doi.org/10.1177/0013164495055003002
21. Gorsuch, R. L. (1974). Factor analysis. Toronto: W. B. Saunders Company.
22. Green, S. B., Levy, R., Thompson, M. S., Lu, M., & Lo, W.-J. (2012). A proposed solution to the problem with using completely random data to assess the number of factors with parallel analysis. Educational and Psychological Measurement, 72(3), 357–374. https://doi.org/10.1177/0013164411422252
23. Green, S. B., Redell, N., Thompson, M. S., & Levy, R. (2016). Accuracy of revised and traditional parallel analyses for assessing dimensionality with binary data. Educational and Psychological Measurement, 76(1), 5–21. https://doi.org/10.1177/0013164415581898
24. Green, S. B., Thompson, M. S., Levy, R., & Lo, W.-J. (2015). Type I and Type II error rates and overall accuracy of the revised parallel analysis method for determining the number of factors. Educational and Psychological Measurement , 75(3), 428-457. https://doi.org/10.1177/0013164414546566
25. Guadagnoli, E., & Velicer, W. F. (1988). Relation of sample size to the stability of component patterns. Psychological Bulletin, 103(2), 265–275.
26. Guilford, J. P. (1952). When not to factor analyze. Psychological Bulletin, 49(1), 26–37. https://doi.org/10.1037/h0054935
27. Harwell, M., Stone, C. A., Hsu, T.-C., & Kirisci, L. (1996). Monte carlo studies in item response theory. Applied Psychological Measurement, 20(2), 101–125. https://doi.org/10.1177/014662169602000201
28. Henson, R. K., & Roberts, J. K. (2006). Use of exploratory factor analysis in published research. Educational and Psychological Measurement, 66(3), 393–416. https://doi.org/10.1177/0013164405282485
29. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179–185. https://doi.org/10.1007/BF02289447
30. Hu, L., Bentler, P. M., & Kano, Y. (1992). Can test statistics in covariance structure analysis be trusted? Psychological Bulletin, 112(2), 351–362. https://doi.org/10.1037/0033-2909.112.2.351
31. Jang, E. E., & Roussos, L. (2007). An investigation into the dimensionality of TOEFL using conditional covariance-based nonparametric approach. Journal of Educational Measurement, 44(1), 1–21. https://doi.org/10.1111/j.1745-3984.2007.00024.x
32. Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20(1), 141–151. https://doi.org/10.1177/001316446002000116
33. Kaya Kalkan, Ö., & Kelecioğlu, H. (2016). The effect of sample size on parametric and nonparametric factor analytical methods. Educational Sciences: Theory & Practice, 16(1), 153–171. https://doi.org/10.12738/estp.2016.1.0220
34. Kim, H. R. (1996). A new index of dimensionality-DETECT. The Pure and Applied Mathematics, 3(2), 141–154.
35. Ledesma, R. D., & Valero-Mora, P. (2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out parallel analysis. Practical Assessment, Research & Evaluation, 12(2), 2–11.
36. Lissitz, R. W., Hou, X., & Slater, S. C. (2012). The contribution of constructed response items to large scale assessment: Measuring and understanding their impact. Journal of Applied Testing Technology, 13(3), 1–50. Retrieved from http://www.jattjournal.com/index.php/atp/article/view/48366/39234
37. MEB. (2019). Sınavla öğrenci alacak ortaöğretim kurumlarına ilişkin merkezî sınav başvuru ve uygulama kılavuzu. Retrieved from https://www.meb.gov.tr/meb_iys_dosyalar/2019_04/03134315_Kilavuz2019.pdf
38. Nandakumar, R., & Stout, W. (1993). Refinements of Stout’s procedure for assessing latent trait unidimensionality. Journal of Educational Statistics, 18(1), 41-68. https://doi.org/10.2307/1165182
39. O’connor, B. P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instruments, & Computers, 32(3), 396–402. https://doi.org/10.3758/BF03200807
40. Osborne, J. W., & Banjanovic, E. S. (2016). Exploratory factor analysis with SAS®. Cary, NC: SAS Intitute Inc.
41. R Core Team. (2018). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.r-project.org/.
42. Raiche, G. (2010). nFactors: An R package for parallel analysis and non graphical solutions to the Cattell scree test. Retrived from https://cran.r-project.org/web/packages/nFactors/nFactors.pdf.
43. Raîche, G., Walls, T. A., Magis, D., Riopel, M., & Blais, J.-G. (2013). Non-Graphical solutions for Cattell’s scree test. Methodology, 9(1), 23–29. https://doi.org/10.1027/1614-2241/a000051
44. Revelle, W. (2016). psych: Procedures for psychological, psychometric, and personality research. Evanston, Illinois. Retrieved from https://cran.r-project.org/package=psych
45. Robey, R. R., & Barcikowski, R. S. (1992). Type I error and the number of iterations in monte carlo studies of robustness. British Journal of Mathematical and Statistical Psychology, 45, 283–288.
46. Robitzsch, A. (2017). sirt: Supplementary item response theory models. Retrieved from https://cran.r-project.org/package=sirt
47. Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52(4), 589–617. https://doi.org/10.1007/BF02294821
48. Streiner, D. L. (1994). Figuring out factors: The use and misuse of factor analysis. Canadian Journal of Psychiatry, 39(3), 135–140.
49. Tabachnik, B. G., & Fidell, L. S. (2012). Using multivariate statistics (6th ed.). Boston: Pearson.
50. van Abswoude, A. A. H., van der Ark, L. A., & Sijtsma, K. (2004). A comparative study of test data dimensionality assessment procedures under nonparametric IRT models. Applied Psychological Measurement, 28(1), 3–24. https://doi.org/10.1177/0146621603259277
51. van den Bergh, H. (1990). On the construct validity of multiple- choice items for reading comprehension. Applied Psychological Measurement, 14(1), 1–12. https://doi.org/10.1177/014662169001400101
52. Velicer, W. F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41(3), 321–327.
53. Weng, L.-J., & Cheng, C.-P. (2005). Parallel analysis with unidimensional binary data. Educational and Psychological Measurement, 65(5), 697–716. https://doi.org/10.1177/0013164404273941
54. Xia, Y., Green, S. B., Xu, Y., & Thompson, M. S. (2018). Proportion of indicator common variance due to a factor as an effect size statistic in revised parallel analysis. Educational and Psychological Measurement. 79(1), 75-107. https://doi.org/10.1177/0013164418754611
55. Yang, Y., & Xia, Y. (2015). On the number of factors to retain in exploratory factor analysis for ordered categorical data. Behavior Research Methods, 47(3), 756–772. https://doi.org/10.3758/s13428-014-0499-2
56. Zhang, J. (2007). Conditional covariance theory and DETECT for polytomous items. Psychometrika, 72(1), 69–91. https://doi.org/10.1007/s11336-004-1257-7
57. Zhang, J., & Stout, W. (1999). The theoretical detect index of dimensionality and its application to approximate simple structure. Psychometrika, 64(2), 213–249. https://doi.org/10.1007/BF02294536
58. Zhang, Y. O., Yu, F., & Nandakumar, R. (2003). The impact of conditional scores on the performance of DETECT. Paper presented at the annual meeting of the National Council on Measurement in Education. Chicago, IL. Retrieved from https://files.eric.ed.gov/fulltext/ED478170.pdf
59. Zoski, K., & Jurs, S. (1993). Using multiple regression to determine the number of factors to retain in factor analysis. Multiple Linear Regression Viewpoint, 20(1), 5–9. Retrieved from http://www.glmj.org/archives/MLRV_1993_20_1.pdf
60. Zoski, K., & Jurs, S. (1996). An objective counterpart to the visual scree test for factor analysis: The standard error scree. Educational and Psychological Measurement, 56(3), 443–451. https://doi.org/10.1177/0013164496056003006
61. Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99(3), 432–442. https://doi.org/10.1037/0033-2909.99.3.432