The Effects of Missing Data Handling Methods on Reliability Coefficients: A Monte Carlo Simulation Study

Year 2024, Volume: 15 Issue: 2, 166 - 182, 30.06.2024

Tugay Kaçak , Abdullah Faruk Kılıç

https://doi.org/10.21031/epod.1485482

Abstract

This study holds significant implications as it examines the impact of different missing data handling methods on the internal consistency coefficients. Using Monte Carlo simulations, we manipulated the number of items, true reliability, sample size, missing data ratio, and mechanisms to compare the relative bias of reliability coefficients. The reliability coefficients under scrutiny in this study encompass Cronbach's Alpha, Heise & Bohrnsted's Omega, Hancock & Mueller's H, Gölbaşı-Şimşek & Noyan's Theta G, Armor's Theta, and Gilmer-Feldt coefficients. Our arsenal of techniques includes single imputation methods like zero, mean, median, and regression imputation, as well as multiple imputation approaches like expectation maximization and random forest. We also employ the classic deletion method known as listwise deletion. The findings suggest that, for missing completely at random (MCAR) or missing at random (MAR) data, single imputation approaches (excluding zero imputation) may still be preferable to expectation maximization and random forest imputation, thereby underscoring the importance of our research.

Keywords

missing data, reliability coefficients, missing data handling methods

Ethical Statement

Araştırma, simülatif veriler ile gerçekleştirilmesi nedeniyle etik kurul iznine ihtiyaç duyulmamaktaıdr.

References

• Allison, P. D. (2002). Missing data. Sage Publications.
• Armor, D. J. (1974). Theta reliability and factor scaling. In H. Costner (Ed.), Sociological Methodology (pp. 17–50). Jossey-Bass.
• Baraldi, A. N., & Enders, C. K. (2010). An introduction to modern missing data analyses. Journal of School Psychology, 48(1), https://doi.org/10.1016/j.jsp.2009.10.001
• Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13(2), 186–203. https://doi.org/10.1207/s15328007sem1302_2
• Béland, S., Jolani, S., Pichette, F., & Renaud, J.-S. (2018). Impact of simple substitution methods for missing data on Classical test theory difficulty and discrimination. The Quantitative Methods for Psychology, 14(3), 180–192. https://doi.org/10.20982/tqmp.14.3.p180
• Bennett, D. A. (2001). How can I deal with missing data in my study? Australian and New Zealand Journal of Public Health, 25(5), 464–469. https://doi.org/10.1111/j.1467-842X.2001.tb00294.x
• Brown, T. A. (2006). Confirmatory factor analysis for applied research. Guilford Press.
• Cheema, J. R. (2014). Some general guidelines for choosing missing data handling methods in educational research. Journal of Modern Applied Statistical Methods, 13(2), 53–75. https://doi.org/10.22237/jmasm/1414814520
• Cho, E. (2023). reliacoef: Compute and compare unidimensional and multidimensional reliability coefficients (1.0.0) [R].
• Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), https://doi.org/10.1007/BF02310555
• Dai, S. (2021). Handling missing responses in psychometrics: Methods and software. Psych, 3(4), 673–693. https://doi.org/10.3390/psych3040043
• Dai, T., Du, Y., Cromley, J., Fechter, T., & Nelson, F. (2024). Analytic approaches to handle missing data in simple matrix sampling planned missing designs. The Journal of Experimental Education, 92(3), 531–558. https://doi.org/10.1080/00220973.2023.2196678
• Doove, L. L., Van Buuren, S., & Dusseldorp, E. (2014). Recursive partitioning for missing data imputation in the presence of interaction effects. Computational Statistics & Data Analysis, 72, 92–104. https://doi.org/10.1016/j.csda.2013.10.025
• Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105 3, 399–412. https://doi.org/10.1111/bjop.12046
• Edwards, A. A., Joyner, K. J., & Schatschneider, C. (2021). A simulation study on the performance of different reliability estimation methods. Educational and Psychological Measurement, 81(6), 1089–1117. https://doi.org/10.1177/0013164421994184
• Enders, C. K. (2003). Using the expectation maximization algorithm to estimate coefficient alpha for scales with item-level missing data. Psychological Methods, 8(3), 322–337. https://doi.org/10.1037/1082-989X.8.3.322
• Enders, C. K. (2004). The impact of missing data on sample reliability estimates: Implications for reliability reporting practices. Educational and Psychological Measurement, 64, 419–436. https://doi.org/10.1177/0013164403261050
• Enders, C. K. (2010). Applied missing data analysis. Guilford Press.
• Fan, J., & Wu, W. (2022). A comparison of multiple imputation strategies to deal with missing nonnormal data in structural equation modeling. Behavior Research Methods. https://doi.org/10.3758/s13428-022-01936-y
• Feldt, L. S., & Charter, R. A. (2003). Estimation of internal consistency reliability when test parts vary in effective length. Measurement and Evaluation in Counseling and Development, 36(1), 23–27. https://doi.org/10.1080/07481756.2003.12069077
• Finch, W. H. (2016). Missing data and multiple imputation in the context of multivariate analysis of variance. The Journal of Experimental Education, 84(2), 356–372. https://doi.org/10.1080/00220973.2015.1011594
• Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466–491. https://doi.org/10.1037/1082-989X.9.4.466
• Gölbaşı-Şimşek, G., & Noyan, F. (2013). McDonald’s ω t , Cronbach’s α, and Generalized θ for composite reliability of common factors structures. Communications in Statistics - Simulation and Computation, 42(9), 2008–2025. https://doi.org/10.1080/03610918.2012.689062
• Goretzko, D. (2021). Factor retention in exploratory factor analysis with missing data. Educational and Psychological Measurement, 82, 444–464. https://doi.org/10.1177/00131644211022031
• Goretzko, D., Heumann, C., & Bühner, M. (2020). Investigating parallel analysis in the context of missing data: A simulation study comparing six missing data methods. Educational and Psychological Measurement, 80, 756–774. https://doi.org/10.1177/0013164419893413
• Gorsuch, R. L. (2015). Factor analysis (Classic edition). Routledge, Taylor & Francis Group.
• Graham, J. W. (2012). Missing data. Springer New York. https://doi.org/10.1007/978-1-4614-4018-5
• Graham, J. W., Cumsille, P., & Shevock, A. E. (2013). Methods for handling missing data. In I. B. Weiner (Ed.), Handbook of Psychology, Second Edition (pp. 109–141). http://doi.org/https://doi.org/10.1002/9781118133880.hop202004
• Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable systems. In R. Cudeck, S. du Toit, & D. Sörbom (Eds.), Structural equation modeling: Present and future—A festschrift in honor of Karl Jöreskog (pp. 195–216). Scientific Software International.
• Hayes, A., & Coutts, J. J. (2020). Use omega rather than Cronbach’s alpha for estimating reliability. But... Communication Methods and Measures, 14, 1–24. https://doi.org/10.1080/19312458.2020.1718629
• Hayes, T., & McArdle, J. J. (2017). Should we impute or should we weight? Examining the performance of two CART-based techniques for addressing missing data in small sample research with nonnormal variables. Computational Statistics & Data Analysis, 115, 35–52. https://doi.org/10.1016/j.csda.2017.05.006
• Heise, D. R., & Bohrnstedt, G. W. (1970). Validity, invalidity, and reliability. Sociological Methodology, 2, 104. https://doi.org/10.2307/270785
• Howell, D. C. (2007). The treatment of missing data. In W. Outhwaite & S. Turner, The SAGE Handbook of Social Science Methodology (pp. 212–226). SAGE Publications Ltd. https://doi.org/10.4135/9781848607958.n11
• Lee, D. Y., Harring, J. R., & Stapleton, L. M. (2019). Comparing methods for addressing missingness in longitudinal modeling of panel data. The Journal of Experimental Education, 87(4), 596–615. https://doi.org/10.1080/00220973.2018.1520683
• Lee, H. J., & Huber, J. C. Jr. (2021). Evaluation of multiple imputation with large proportions of missing data: How much is too much? Iranian Journal of Public Health. https://doi.org/10.18502/ijph.v50i7.6626
• Lei, P.-W., & Shiverdecker, L. K. (2020). Performance of estimators for confirmatory factor analysis of ordinal variables with missing data. Structural Equation Modeling: A Multidisciplinary Journal, 27(4), 584–601. https://doi.org/10.1080/10705511.2019.1680292
• Leite, W., & Beretvas, S. N. (2010). The performance of multiple imputation for likert-type items with missing data. Journal of Modern Applied Statistical Methods, 9(1), 64–74. https://doi.org/10.22237/jmasm/1272686820
• Li, J., & Lomax, R. G. (2017). Effects of missing data methods in SEM under conditions of incomplete and nonnormal Data. The Journal of Experimental Education, 85(2), 231–258. https://doi.org/10.1080/00220973.2015.1134418
• Little, R., & Rubin, D. (2002). Statistical analysis with missing data (1st ed.). Wiley. https://doi.org/10.1002/9781119013563
• Little, R., & Rubin, D. (2019). Statistical analysis with missing data (3rd ed.). Wiley. https://doi.org/10.1002/9781119482260
• McAllister, D. J., & Bigley, G. A. (2002). Work context and the definition of self: How organizational care influences organization-based self-esteem. Academy of Management Journal, 45(5), 894–904. https://doi.org/10.2307/3069320
• McDonald, R. P. (1970). The theoretical foundations of principal factor analysis, canonical factor analysis, and alpha factor analysis. British Journal of Mathematical and Statistical Psychology, 23(1), 1–21. https://doi.org/10.1111/j.2044-8317.1970.tb00432.x
• McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum Associates.
• McNeish, D. M. (2017). Exploratory factor analysis with small samples and missing data. Journal of Personality Assessment, 99(6), 637–652. https://doi.org/10.1080/00223891.2016.1252382
• McNeish, D. M. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/10.1037/met0000144
• Moritz, S., & Bartz-Beielstein, T. (2017). imputeTS: Time Series Missing Value Imputation in R. The R Journal, 9(1), 207. https://doi.org/10.32614/RJ-2017-009
• Myers, T. A. (2011). Goodbye, listwise deletion: Presenting hot deck imputation as an easy and effective tool for handling missing data. Communication Methods and Measures, 5(4), 297–310. https://doi.org/10.1080/19312458.2011.624490
• Newman, D. A. (2014). Missing data. Organizational Research Methods. https://doi.org/10.1177/1094428114548590
• Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3nd ed.). McGraw-Hill.
• Parent, M. C. (2013). Handling item-level missing data: Simpler is just as good. The Counseling Psychologist, 41(4), 568–600. https://doi.org/10.1177/0011000012445176
• Revelle, W. (2024). psych: Procedures for psychological, psychometric, and personality research (R package version 2.4.1) [Computer software]. https://CRAN.R-project.org/package=psych
• Rockel, T. (2022). missMethods: Methods for missing data (0.4.0) [R]. https://github.com/torockel/missMethods
• Roth, P. L., Switzer, F. S., & Switzer, D. M. (1999). Missing data in multiple item scales: A monte carlo analysis of missing data techniques. Organizational Research Methods, 2(3), 211–232. https://doi.org/10.1177/109442819923001
• Şahin Kürşad, M., & Nartgün, Z. (2015). Kayıp veri sorununun çözümünde kullanılan farklı yöntemlerin ölçeklerin geçerlik ve güvenirliği bağlamında karşılaştırılması. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 6(2), 254–267. https://doi.org/10.21031/epod.95917
• Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), https://doi.org/10.1037/1082-989X.7.2.147
• Scheffer, J. (2002). Dealing with missing data (1st ed.). Massey University. https://mro.massey.ac.nz/handle/10179/4355
• Shah, A. D., Bartlett, J. W., Carpenter, J., Nicholas, O., & Hemingway, H. (2014). Comparison of random forest and parametric imputation models for imputing missing data using MICE: A CALIBER study. American Journal of Epidemiology, 179(6), 764–774. https://doi.org/10.1093/aje/kwt312
• Sheng, Y., & Sheng, Z. (2012). Is coefficient alpha robust to non-normal data? Frontiers in Psychology, 3(34). https://doi.org/10.3389/fpsyg.2012.00034
• Sigal, M. J., & Chalmers, R. P. (2016). Play it again: Teaching statistics with monte carlo simulation. Journal of Statistics Education, 24(3), 136–156. https://doi.org/10.1080/10691898.2016.1246953
• Sijtsma, K., & Van Der Ark, L. A. (2003). Investigation and treatment of missing item scores in test and questionnaire data. Multivariate Behavioral Research, 38(4), 505–528. https://doi.org/10.1207/s15327906mbr3804_4
• Trizano-Hermosilla, I., & Alvarado, J. M. (2016). Best alternatives to Cronbach’s Alpha reliability in realistic conditions: Congeneric and asymmetrical measurements. Frontiers in Psychology, 7. https://doi.org/10.3389/fpsyg.2016.00769
• Turner, H. J., Natesan, P., & Henson, R. K. (2017). Performance Evaluation of confidence intervals for ordinal coefficient alpha. Journal of Modern Applied Statistical Methods, 16(2), 157–185. https://doi.org/10.22237/jmasm/1509494940
• Uysal, İ., & Kılıç, A. (2022). Normal dağılım ikilemi. Anadolu Journal of Educational Sciences International, 12(1), 220–248. https://doi.org/10.18039/ajesi.962653
• van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software, 45(3). https://doi.org/10.18637/jss.v045.i03
• Van Ginkel, J. R., Van Der Ark, L. A., & Sijtsma, K. (2007). Multiple imputation of item scores in test and questionnaire data, and influence on psychometric results. Multivariate Behavioral Research, 42(2), 387–414. https://doi.org/10.1080/00273170701360803
• Wei, R., Wang, J., Su, M., Jia, E., Chen, S., Chen, T., & Ni, Y. (2018). Missing value imputation approach for mass spectrometry-based metabolomics data. Scientific Reports, 8(1), 663. https://doi.org/10.1038/s41598-017-19120-0
• Zhang, Z. (2016). Missing data imputation: Focusing on single imputation. Annals of Translational Medicine, 4(1), 1–8. https://doi.org/10.3978/j.issn.2305-5839.2015.12.38
• Zhang, Z., & Yuan, K.-H. (2016). Robust coefficients alpha and omega and confidence intervals with outlying observations and missing data: Methods and software. Educational and Psychological Measurement, 76(3), 387–411. https://doi.org/10.1177/0013164415594658