Comparison of Confirmatory Factor Analysis Estimation Methods on Binary Data

Abstract

This Monte Carlo simulation study aimed to investigate confirmatory factor analysis (CFA) estimation methods under different conditions, such as sample size, distribution of indicators, test length, average factor loading, and factor structure. Binary data were generated to compare the performance of maximum likelihood (ML), mean and variance adjusted unweighted least squares (ULSMV), mean and variance adjusted weighted least squares (WLSMV), and Bayesian estimators. As a result of the study, it was revealed that increased average factor loading and sample size had a positive effect on the performance of the estimation methods. According to the research findings, it can be said that the methods are sufficient to estimate average factor loading and interfactor correlations, regardless of the estimation methods, in most of the conditions where the average factor loading is 0.7. In small sample sizes particularly, the interfactor correlation was underestimated for skewed indicator conditions. According to the findings of the study, although there is not the most accurate method in all conditions, it can be recommended to use ULSMV method because it performs adequately in more conditions.

Keywords

• Confirmatory factor analysis
• Estimation methods
• Binary data
• Simulation

References

1. Acar-Güvendir, M., & Özer-Özkan, Y. (2015). Türkiye’deki eğitim alanında yayımlanan bilimsel dergilerde ölçek geliştirme ve uyarlama konulu makalelerin incelenmesi [The examination of scale development and scale adaptation articles published in Turkish academic journals on education]. Electronic Journal of Social Sciences, 14(52), 23–33. https://doi.org/10.17755/esosder.54872
2. AERA, APA, & NCME. (2014). Standards for educational and psychological testing. American Educational Research Association.
3. Anıl, D., Güzeller, C. O., Çokluk, Ö., & Şekercioǧlu, G. (2010). Level determination exam (SBS-2008) the determination of the validity and reliability of 7th grade mathematics sub- test. Procedia Social and Behavioral Sciences, 2(2), 5292 5298. https://doi.org/10.1016/j.sbspro.2010.03.863
4. Babakus, E., Ferguson, C. E., & Jöreskog, K. G. (1987). The sensitivity of confirmatory maximum likelihood factor analysis to violations of measurement scale and distributional assumptions. Journal of Marketing Research, 24(2), 222 228. https://doi.org/10.2307/3151512
5. Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13(2), 186 203. https://doi.org/10.1207/s15328007sem1302_2
6. Bollen, K. A. (1989). Structural equations with latent variables. John Wiley & Sons, Inc. https://doi.org/10.1002/9781118619179
7. Boomsma, A. (1985). Nonconvergence, improper solutions, and starting values in lisrel maximum likelihood estimation. Psychometrika, 50(2), 229 242. https://doi.org/10.1007/BF02294248
8. Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). The Guilford.

9. Byrne, B. M. (2016). Structural equation modeling with AMOS: Basic concepts, applications, and programming (3rd ed.). Routledge.
10. Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6(4), 330–351. https://doi.org/10.1037/1082-989X.6.4.330
11. Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis (2nd ed.). Lawrence Erlbaum Associates.
12. Crocker, L., & Algina, J. (2008). Introduction of classical and modern test theory. Cengage Learning.
13. Curran, P. J., West, S. G., & Finch, J. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16–29. https://doi.org/10.1037/1082-989X.1.1.16
14. DiStefano, C., & Morgan, G. B. (2014). A comparison of diagonal weighted least squares robust estimation techniques for ordinal data. Structural Equation Modeling: A Multidisciplinary Journal, 21(3), 425 438. https://doi.org/10.1080/10705511.2014.915373
15. Dolan, C. V. (1994). Factor analysis of variables with 2, 3, 5 and 7 response categories: A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology, 47(2), 309 326. https://doi.org/10.1111/j.2044-8317.1994.tb01039.x
16. Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466–491. https://doi.org/10.1037/1082-989X.9.4.466
17. Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7(3), 286–299. https://doi.org/10.1037/1040-3590.7.3.286
18. Forero, C. G., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A monte carlo study comparing DWLS and ULS estimation. Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 625 641. https://doi.org/10.1080/10705510903203573
19. Gilbert, N. (1999). Simulation: A new way of doing social science. American Behavioral Scientist, 42(10), 1485–1487. https://doi.org/10.1177/0002764299042010002
20. Gorsuch, R. L. (1974). Factor analysis. W. B. Saunders.
21. Green, S. B., Akey, T. M., Fleming, K. K., Hershberger, S. L., & Marquis, J. G. (1997). Effect of the number of scale points on chi‐square fit indices in confirmatory factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 4(2), 108 120. https://doi.org/10.1080/10705519709540064
22. Guadagnoli, E., & Velicer, W. F. (1988). Relation of sample size to the stability of component patterns. Psychological Bulletin, 103(2), 265 275. https://doi.org/10.1037/0033 2909.103.2.265
23. Hallquist, M., & Wiley, J. (2017). MplusAutomation: Automating Mplus Model Estimation and Interpretation [Computer software]. https://cran.r project.org/web/packages/MplusAutomation/
24. Harrison, R. L. (2010). Introduction to Monte Carlo Simulation. AIP Conference Proceedings, 1204, 17–21. https://doi.org/10.1063/1.3295638
25. Holtmann, J., Koch, T., Lochner, K., & Eid, M. (2016). A Comparison of ML, WLSMV, and Bayesian methods for multilevel structural equation models in small samples: A simulation study. Multivariate Behavioral Research, 51(5), 661 680. https://doi.org/10.1080/00273171.2016.1208074
26. Jin, S., Luo, H., & Yang-Wallentin, F. (2016). A simulation study of polychoric instrumental variable estimation in structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 23(5), 680 694. https://doi.org/10.1080/10705511.2016.1189334
27. Kılıç, A. F., & Kelecioğlu, H. (2016). TEOG ortak ve mazeret sınavındaki Türkçe ve matematik alt testlerinin psikometrik özelliklerinin karşılaştırılması [The comparison of psychometric properties of standardised and make up maths and Turkish subtest questions in TEOG]. Journal of Measurement and Evaluation in Education and Psychology, 7(1), 33–58. https://doi.org/10.21031/epod.14532
28. Kline, R. B. (2016). Principle and practice of structural equation modeling (4th ed.). The Guilford.
29. Koğar, H., & Yılmaz Koğar, E. (2015). Comparison of different estimation methods for categorical and ordinal data in confirmatory factor analysis. Journal of Measurement and Evaluation in Education and Psychology, 6(2), 351 364. https://doi.org/10.21031/epod.94857
30. Lei, P. (2009). Evaluating estimation methods for ordinal data in structural equation modeling. Quality and Quantity, 43(3), 495–507. https://doi.org/10.1007/s11135-007-9133-z
31. Li, C.-H. (2016a). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48(3), 936–949. https://doi.org/10.3758/s13428-015-0619-7
32. Li, C.-H. (2016b). The performance of ML, DWLS, and ULS estimation with robust corrections in structural equation models with ordinal variables. Psychological Methods, 21(3), 369–387. https://doi.org/10.1037/met0000093
33. Liang, X., & Yang, Y. (2014). An evaluation of WLSMV and Bayesian methods for confirmatory factor analysis with categorical indicators. International Journal of Quantitative Research in Education, 2(1), 17 38. https://doi.org/10.1504/IJQRE.2014.060972
34. Lissitz, R. W., Hou, X., & Slater, S. C. (2012). The contribution of constructed response items to large scale assessment: Measuring and understanding their impact. Journal of Applied Testing Technology, 13(3), 1 50. http://www.jattjournal.com/index.php/atp/article/view/48366/39234
35. Morata-Ramirez, M. de los A., & Holgado-Tello, F. P. (2013). Construct validity of likert scales through confirmatory factor analysis: A simulation study comparing different methods of estimation based on Pearson and polychoric correlations. International Journal of Social Science Studies, 1(1), 54-61. https://doi.org/10.11114/ijsss.v1i1.27
36. Moshagen, M., & Musch, J. (2014). Sample size requirements of the robust weighted least squares estimator. Methodology, 10(2), 60 70. https://doi.org/10.1027/1614 2241/a000068
37. Mulaik, S. A. (2009). Linear causal modeling with structural equations. Chapman & Hall.
38. Muthén, B. O., & Asparouhov, T. (2002). Latent variable analysis with categorical outcomes: Multiple group and growth modeling in Mplus. https://www.statmodel.com/download/webnotes/CatMGLong.pdf
39. Muthén, B. O., & Asparouhov, T. (2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17(3), 313–335. https://doi.org/10.1037/a0026802
40. Muthén, B. O., & Kaplan, D. (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38(2), 171–189. https://doi.org/10.1111/j.2044-8317.1985.tb00832.x
41. Muthén, L. K., & Muthén, B. O. (2012). Mplus statistical modeling software: Release 7.0 [Computer software]. Muthén & Muthén.
42. Muthén, L. K., & Muthén, B. O. (2015). Mplus user’s guide (7th ed.). Muthén & Muthén.
43. Nalbantoğlu Yılmaz, F. (2019). Comparison of different estimation methods used in confirmatory factor analyses in non-normal data: A monte carlo study. International Online Journal of Educational Sciences, 11(4), 131 140. https://doi.org/10.15345/iojes.2019.04.010
44. Nestler, S. (2013). A monte carlo study comparing PIV, ULS and DWLS in the estimation of dichotomous confirmatory factor analysis. British Journal of Mathematical and Statistical Psychology, 66(1), 127 143. https://doi.org/10.1111/j.2044 8317.2012.02044.x
45. Önen, E. (2019). A comparison of frequentist and Bayesian approaches: The power to detect model misspecifications in confirmatory factor analytic models. Universal Journal of Educational Research, 7(2), 494–514. https://doi.org/10.13189/ujer.2019.070223
46. Potthast, M. J. (1993). Confirmatory factor analysis of ordered categorical variables with large models. British Journal of Mathematical and Statistical Psychology, 46(2), 273–286. https://doi.org/10.1111/j.2044-8317.1993.tb01016.x
47. R Core Team. (2018). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.r-project.org/
48. Raykov, T., & Marcoulides, G. A. (2006). A first course in structural equation modeling (2nd ed.). Lawrence Erlbaum Associates.
49. Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373. https://doi.org/10.1037/a0029315
50. Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
51. Şahin, M. G., & Boztunç Öztürk, N. (2018). Eğitim alanında ölçek geliştirme süreci: Bir içerik analizi çalışması [Scale development process in educational field: A content analysis research]. Kastamonu Education Journal, 26(1), 191 199. https://doi.org/10.24106/kefdergi.375863
52. Shi, D., DiStefano, C., McDaniel, H. L., & Jiang, Z. (2018). Examining chi-square test statistics under conditions of large model size and ordinal Data. Structural Equation Modeling: A Multidisciplinary Journal, 25(6), 924 945. https://doi.org/10.1080/10705511.2018.1449653
53. Stevens, J. P. (2009). Applied multivariate statistics for the social science (5th ed.). Taylor & Francis.
54. Streiner, D. L. (1994). Figuring out factors: The use and misuse of factor analysis. Canadian Journal of Psychiatry, 39(3), 135 140. https://doi.org/10.1177%2F070674379403900303
55. Tabachnik, B. G., & Fidell, L. S. (2012). Using multivariate statistics (6th ed.). Pearson.
56. Thissen, D., Wainer, H., & Wang, X.-B. (1994). Are tests comprising both multiple-choice and free-response items necessarily less unidimensional than multiple-choice tests? An analysis of two tests. Journal of Educational Measurement, 31(2), 113–123. https://doi.org/10.1111/j.1745-3984.1994.tb00437.x
57. Wolf, E. J., Harrington, K. M., Clark, S. L., & Miller, M. W. (2013). Sample size requirements for structural equation models: An evaluation of power, bias, and solution propriety. National Institutes of Health, 76(6), 913 934. https://doi.org/10.1177/0013164413495237
58. Xu, M. (2019). A comparison of frequentist and Bayesian approaches for confirmatory factor analysis (Publication No. 27534819) [Doctoral dissertation, The Ohio State University]. ProQuest Dissertations and Theses Global.
59. Yang-Wallentin, F., Jöreskog, K. G., & Luo, H. (2010). Confirmatory factor analysis of ordinal variables with misspecified models. Structural Equation Modeling: A Multidisciplinary Journal, 17(3), 392–423. https://doi.org/10.1080/10705511.2010.489003