Investigation of the Effect of Missing Data Handling Methods on Measurement Invariance of Multi-Dimensional Structures

Abstract

The purpose of this study was to compare the missing data handling methods on measurement invariance of multi-dimensional structures. For this purpose, data of 10857 students who participated in PISA 2015 administration from Turkey and Singapore and fully responded to the items related to affective characteristics of science literacy was used. Data with different percentages of missing data (5%, 10%, and 20% missing data) were generated from the complete data set with missing completely at random (MCAR) mechanism. In all data sets, missing data was completed with listwise deletion (LD), serial mean imputation (SMI), regression imputation (RI), expectation maximization (EM), and multiple imputation (MI) methods. Measurement invariance of the construct being measured between countries on completed data sets was investigated with multiple-group confirmatory factor analysis (MG-CFA). Findings from each dataset were compared with reference values. In the results of the study, RI and MI methods in the data set with 5% missing, EM method in the data set with 10% missing, and MI method in the data set with 20% missing gave the more similar results to the reference values than the other methods.

Keywords

• missing data handling methods
• measurement invariance
• multiple-group confirmatory factor analysis
• PISA 2015
• science literacy

References

1. Akbaş, U., & Tavşancıl, E. (2015). Farklı örneklem büyüklüklerinde ve kayıp veri örüntülerinde ölçeklerin psikometrik özelliklerinin kayıp veri baş etme teknikleri ile incelenmesi. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 6(1), 38-57.
2. Alıcı, D. (2013). Okula yönelik tutum ölçeği’nin geliştirilmesi: Güvenirlik ve geçerlik çalışması. Eğitim ve Bilim, 38(268), 318-331.
3. Allison, P. D. (2001). Missing data. Thousand Oaks, CA: Sage.
4. Allison, P. D. (2003). Missing data techniques for structural equation modeling. Journal of Abnormal Psychology, 112(4), 545-557.
5. Asparouhov, T., & Muthen, B. (2014). Multiple-group factor analysis alignment. Structural Equation Modeling: A Multidisciplinary Journal, 21(4), 1-14.
6. Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: The Guilford Press.
7. Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464-504.
8. Chen, S. F., Wang, S., & Chen, C. Y. (2012). A simulation study using EFA and CFA programs based the impact of the missing data on test dimensionality. Expert Systems with Applications, 39(4), 4026-4031.

9. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233-255.
10. Chung, H., Kim, J., Park, R., Bamer, A. M., Bocell, F. D., & Amtmann, D. (2016). Testing the measurement invariance of the University of Washington Self-Efficacy Scale short form across four diagnostic subgroups. Qual Life Res., 25(10), 2559-2564.
11. Çüm, S. & Gelbal, S. (2015). Kayıp veriler yerine yaklaşık değer atamada kullanılan farklı yöntemlerin model veri uyumu üzerindeki etkisi. Mehmet Akif Ersoy Üniversitesi Eğitim Fakültesi Dergisi, 35, 87-111.
12. Demir, E. (2013). Kayıp verilerin varlığında çoktan seçmeli testlerde madde ve test parametrelerinin kestirilmesi: SBS örneği. Eğitim Bilimleri Araştırmaları Dergisi, 3(2), 47-68.
13. Downey, R. G. & King, C. V. (1998). Missing data in likert ratings: A comparison of replacement methods. The Journal of General Psychology, 125(2), 175-191.
14. Drasgow, F. (1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are the central issues. Psychological Bulletin, 95(1), 134-135.
15. Enders, C. K. (2010). Applied missing data analysis. New York: The Guilford Press.
16. Harrington, D. (2009). Confirmatory factor analysis. New York: Oxford University Press.
17. Köse, A. (2014). The effect of missing data handling methods on goodness of fit indices in confirmatory factor analysis. Educational Research and Reviews, 9(8), 208-215.
18. Little, R. J. A. & Rubin, D. B. (2002). Statistical analysis with missing data. (2nd edition). New York: Wiley
19. OECD (Organization for Economic Cooperation and Development) (2016). PISA 2015 results in focus. Retrieved February 12, 2017, from https://www.oecd.org/pisa/pisa-2015-results-in-focus.pdf
20. Olinsky, A., Chen, S., & Harlow, L. (2003). The comparative efficacy of imputation methods for missing data in structural equation modeling. European Journal of Operational Research, 151(1), 53-79.
21. Reise, S. P., Widaman, K. F., & Pugh, R. H. (1993). Confirmatory factor analysis and item response theory: Two approaches for exploring measurement invariance. Psychological Bulletin, 114(3), 552-566.
22. Schafer, J. L. & Graham, J. W. (2002) Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147-177.
23. Schnabel, D. B. L., Kelava, A., Vijver, F. J. R., & Seifert, L. (2015). Examining psychometric properties, measurement invariance, and construct validity of a short version of the Test to Measure Intercultural Competence (TMIC-S) in Germany and Brazil. International Journal of Intercultural Relations, 49, 137-155.
24. Schoot, R., Lugtig, P., & Hox, J. (2012). A checklist for testing measurement invariance. European Journal of Developmental Psychology, 9(4), 486-492.
25. Selvi, H., Alıcı, D., & Uzun, N. B. (2020). Investigating measurement invariance under different missing value reduction methods. Asian Journal of Education and Training, 6(2), 237-245.
26. Tabachnick, B. G. & Fidell, L. S. (2013). Using multivariate statistics. (6th edition). Boston: Pearson.
27. Wang, M., Willett, J. B., & Eccles, J. S. (2011). The assessment of school engagement: Examining dimensionality and measurement invariance by gender and race/ethnicity. Journal of School Psychology, 49(4), 465-480.
28. Whitaker, B. G., & Mckinney, J. L. (2007). Assessing the measurement invariance of latent job satisfaction ratings across survey administration modes for respondent subgroups: A MIMIC modeling approach. Behavior Research Methods, 39(3), 502-509.
29. Xu, H., & Tracey, T. J. G. (2017). Use of multi-group confirmatory factor analysis in examining measurenet invariance in counseling psychology research. The European Journal of Counselling Psychology, 6(1), 75-82.