The Effects of Missing Data Handling Methods on Reliability Coefficients: A Monte Carlo Simulation Study

Abstract

This study holds significant implications as it examines the impact of different missing data handling methods on the internal consistency coefficients. Using Monte Carlo simulations, we manipulated the number of items, true reliability, sample size, missing data ratio, and mechanisms to compare the relative bias of reliability coefficients. The reliability coefficients under scrutiny in this study encompass Cronbach's Alpha, Heise & Bohrnsted's Omega, Hancock & Mueller's H, Gölbaşı-Şimşek & Noyan's Theta G, Armor's Theta, and Gilmer-Feldt coefficients. Our arsenal of techniques includes single imputation methods like zero, mean, median, and regression imputation, as well as multiple imputation approaches like expectation maximization and random forest. We also employ the classic deletion method known as listwise deletion. The findings suggest that, for missing completely at random (MCAR) or missing at random (MAR) data, single imputation approaches (excluding zero imputation) may still be preferable to expectation maximization and random forest imputation, thereby underscoring the importance of our research.

Keywords

• missing data
• reliability coefficients
• missing data handling methods

References

1. Allison, P. D. (2002). Missing data. Sage Publications.
2. Armor, D. J. (1974). Theta reliability and factor scaling. In H. Costner (Ed.), Sociological Methodology (pp. 17–50). Jossey-Bass.
3. Baraldi, A. N., & Enders, C. K. (2010). An introduction to modern missing data analyses. Journal of School Psychology, 48(1), https://doi.org/10.1016/j.jsp.2009.10.001
4. Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13(2), 186–203. https://doi.org/10.1207/s15328007sem1302_2
5. Béland, S., Jolani, S., Pichette, F., & Renaud, J.-S. (2018). Impact of simple substitution methods for missing data on Classical test theory difficulty and discrimination. The Quantitative Methods for Psychology, 14(3), 180–192. https://doi.org/10.20982/tqmp.14.3.p180
6. Bennett, D. A. (2001). How can I deal with missing data in my study? Australian and New Zealand Journal of Public Health, 25(5), 464–469. https://doi.org/10.1111/j.1467-842X.2001.tb00294.x
7. Brown, T. A. (2006). Confirmatory factor analysis for applied research. Guilford Press.
8. Cheema, J. R. (2014). Some general guidelines for choosing missing data handling methods in educational research. Journal of Modern Applied Statistical Methods, 13(2), 53–75. https://doi.org/10.22237/jmasm/1414814520

9. Cho, E. (2023). reliacoef: Compute and compare unidimensional and multidimensional reliability coefficients (1.0.0) [R].
10. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), https://doi.org/10.1007/BF02310555
11. Dai, S. (2021). Handling missing responses in psychometrics: Methods and software. Psych, 3(4), 673–693. https://doi.org/10.3390/psych3040043
12. Dai, T., Du, Y., Cromley, J., Fechter, T., & Nelson, F. (2024). Analytic approaches to handle missing data in simple matrix sampling planned missing designs. The Journal of Experimental Education, 92(3), 531–558. https://doi.org/10.1080/00220973.2023.2196678
13. Doove, L. L., Van Buuren, S., & Dusseldorp, E. (2014). Recursive partitioning for missing data imputation in the presence of interaction effects. Computational Statistics & Data Analysis, 72, 92–104. https://doi.org/10.1016/j.csda.2013.10.025
14. Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105 3, 399–412. https://doi.org/10.1111/bjop.12046
15. Edwards, A. A., Joyner, K. J., & Schatschneider, C. (2021). A simulation study on the performance of different reliability estimation methods. Educational and Psychological Measurement, 81(6), 1089–1117. https://doi.org/10.1177/0013164421994184
16. Enders, C. K. (2003). Using the expectation maximization algorithm to estimate coefficient alpha for scales with item-level missing data. Psychological Methods, 8(3), 322–337. https://doi.org/10.1037/1082-989X.8.3.322
17. Enders, C. K. (2004). The impact of missing data on sample reliability estimates: Implications for reliability reporting practices. Educational and Psychological Measurement, 64, 419–436. https://doi.org/10.1177/0013164403261050
18. Enders, C. K. (2010). Applied missing data analysis. Guilford Press.
19. Fan, J., & Wu, W. (2022). A comparison of multiple imputation strategies to deal with missing nonnormal data in structural equation modeling. Behavior Research Methods. https://doi.org/10.3758/s13428-022-01936-y
20. Feldt, L. S., & Charter, R. A. (2003). Estimation of internal consistency reliability when test parts vary in effective length. Measurement and Evaluation in Counseling and Development, 36(1), 23–27. https://doi.org/10.1080/07481756.2003.12069077
21. Finch, W. H. (2016). Missing data and multiple imputation in the context of multivariate analysis of variance. The Journal of Experimental Education, 84(2), 356–372. https://doi.org/10.1080/00220973.2015.1011594
22. Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466–491. https://doi.org/10.1037/1082-989X.9.4.466
23. Gölbaşı-Şimşek, G., & Noyan, F. (2013). McDonald’s ω t , Cronbach’s α, and Generalized θ for composite reliability of common factors structures. Communications in Statistics - Simulation and Computation, 42(9), 2008–2025. https://doi.org/10.1080/03610918.2012.689062
24. Goretzko, D. (2021). Factor retention in exploratory factor analysis with missing data. Educational and Psychological Measurement, 82, 444–464. https://doi.org/10.1177/00131644211022031
25. Goretzko, D., Heumann, C., & Bühner, M. (2020). Investigating parallel analysis in the context of missing data: A simulation study comparing six missing data methods. Educational and Psychological Measurement, 80, 756–774. https://doi.org/10.1177/0013164419893413
26. Gorsuch, R. L. (2015). Factor analysis (Classic edition). Routledge, Taylor & Francis Group.
27. Graham, J. W. (2012). Missing data. Springer New York. https://doi.org/10.1007/978-1-4614-4018-5
28. Graham, J. W., Cumsille, P., & Shevock, A. E. (2013). Methods for handling missing data. In I. B. Weiner (Ed.), Handbook of Psychology, Second Edition (pp. 109–141). http://doi.org/https://doi.org/10.1002/9781118133880.hop202004
29. Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable systems. In R. Cudeck, S. du Toit, & D. Sörbom (Eds.), Structural equation modeling: Present and future—A festschrift in honor of Karl Jöreskog (pp. 195–216). Scientific Software International.
30. Hayes, A., & Coutts, J. J. (2020). Use omega rather than Cronbach’s alpha for estimating reliability. But... Communication Methods and Measures, 14, 1–24. https://doi.org/10.1080/19312458.2020.1718629
31. Hayes, T., & McArdle, J. J. (2017). Should we impute or should we weight? Examining the performance of two CART-based techniques for addressing missing data in small sample research with nonnormal variables. Computational Statistics & Data Analysis, 115, 35–52. https://doi.org/10.1016/j.csda.2017.05.006
32. Heise, D. R., & Bohrnstedt, G. W. (1970). Validity, invalidity, and reliability. Sociological Methodology, 2, 104. https://doi.org/10.2307/270785
33. Howell, D. C. (2007). The treatment of missing data. In W. Outhwaite & S. Turner, The SAGE Handbook of Social Science Methodology (pp. 212–226). SAGE Publications Ltd. https://doi.org/10.4135/9781848607958.n11
34. Lee, D. Y., Harring, J. R., & Stapleton, L. M. (2019). Comparing methods for addressing missingness in longitudinal modeling of panel data. The Journal of Experimental Education, 87(4), 596–615. https://doi.org/10.1080/00220973.2018.1520683
35. Lee, H. J., & Huber, J. C. Jr. (2021). Evaluation of multiple imputation with large proportions of missing data: How much is too much? Iranian Journal of Public Health. https://doi.org/10.18502/ijph.v50i7.6626
36. Lei, P.-W., & Shiverdecker, L. K. (2020). Performance of estimators for confirmatory factor analysis of ordinal variables with missing data. Structural Equation Modeling: A Multidisciplinary Journal, 27(4), 584–601. https://doi.org/10.1080/10705511.2019.1680292
37. Leite, W., & Beretvas, S. N. (2010). The performance of multiple imputation for likert-type items with missing data. Journal of Modern Applied Statistical Methods, 9(1), 64–74. https://doi.org/10.22237/jmasm/1272686820
38. Li, J., & Lomax, R. G. (2017). Effects of missing data methods in SEM under conditions of incomplete and nonnormal Data. The Journal of Experimental Education, 85(2), 231–258. https://doi.org/10.1080/00220973.2015.1134418
39. Little, R., & Rubin, D. (2002). Statistical analysis with missing data (1st ed.). Wiley. https://doi.org/10.1002/9781119013563
40. Little, R., & Rubin, D. (2019). Statistical analysis with missing data (3rd ed.). Wiley. https://doi.org/10.1002/9781119482260
41. McAllister, D. J., & Bigley, G. A. (2002). Work context and the definition of self: How organizational care influences organization-based self-esteem. Academy of Management Journal, 45(5), 894–904. https://doi.org/10.2307/3069320
42. McDonald, R. P. (1970). The theoretical foundations of principal factor analysis, canonical factor analysis, and alpha factor analysis. British Journal of Mathematical and Statistical Psychology, 23(1), 1–21. https://doi.org/10.1111/j.2044-8317.1970.tb00432.x
43. McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum Associates.
44. McNeish, D. M. (2017). Exploratory factor analysis with small samples and missing data. Journal of Personality Assessment, 99(6), 637–652. https://doi.org/10.1080/00223891.2016.1252382
45. McNeish, D. M. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/10.1037/met0000144
46. Moritz, S., & Bartz-Beielstein, T. (2017). imputeTS: Time Series Missing Value Imputation in R. The R Journal, 9(1), 207. https://doi.org/10.32614/RJ-2017-009
47. Myers, T. A. (2011). Goodbye, listwise deletion: Presenting hot deck imputation as an easy and effective tool for handling missing data. Communication Methods and Measures, 5(4), 297–310. https://doi.org/10.1080/19312458.2011.624490
48. Newman, D. A. (2014). Missing data. Organizational Research Methods. https://doi.org/10.1177/1094428114548590
49. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3nd ed.). McGraw-Hill.
50. Parent, M. C. (2013). Handling item-level missing data: Simpler is just as good. The Counseling Psychologist, 41(4), 568–600. https://doi.org/10.1177/0011000012445176
51. Revelle, W. (2024). psych: Procedures for psychological, psychometric, and personality research (R package version 2.4.1) [Computer software]. https://CRAN.R-project.org/package=psych
52. Rockel, T. (2022). missMethods: Methods for missing data (0.4.0) [R]. https://github.com/torockel/missMethods
53. Roth, P. L., Switzer, F. S., & Switzer, D. M. (1999). Missing data in multiple item scales: A monte carlo analysis of missing data techniques. Organizational Research Methods, 2(3), 211–232. https://doi.org/10.1177/109442819923001
54. Şahin Kürşad, M., & Nartgün, Z. (2015). Kayıp veri sorununun çözümünde kullanılan farklı yöntemlerin ölçeklerin geçerlik ve güvenirliği bağlamında karşılaştırılması. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 6(2), 254–267. https://doi.org/10.21031/epod.95917
55. Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), https://doi.org/10.1037/1082-989X.7.2.147
56. Scheffer, J. (2002). Dealing with missing data (1st ed.). Massey University. https://mro.massey.ac.nz/handle/10179/4355
57. Shah, A. D., Bartlett, J. W., Carpenter, J., Nicholas, O., & Hemingway, H. (2014). Comparison of random forest and parametric imputation models for imputing missing data using MICE: A CALIBER study. American Journal of Epidemiology, 179(6), 764–774. https://doi.org/10.1093/aje/kwt312
58. Sheng, Y., & Sheng, Z. (2012). Is coefficient alpha robust to non-normal data? Frontiers in Psychology, 3(34). https://doi.org/10.3389/fpsyg.2012.00034
59. Sigal, M. J., & Chalmers, R. P. (2016). Play it again: Teaching statistics with monte carlo simulation. Journal of Statistics Education, 24(3), 136–156. https://doi.org/10.1080/10691898.2016.1246953
60. Sijtsma, K., & Van Der Ark, L. A. (2003). Investigation and treatment of missing item scores in test and questionnaire data. Multivariate Behavioral Research, 38(4), 505–528. https://doi.org/10.1207/s15327906mbr3804_4
61. Trizano-Hermosilla, I., & Alvarado, J. M. (2016). Best alternatives to Cronbach’s Alpha reliability in realistic conditions: Congeneric and asymmetrical measurements. Frontiers in Psychology, 7. https://doi.org/10.3389/fpsyg.2016.00769
62. Turner, H. J., Natesan, P., & Henson, R. K. (2017). Performance Evaluation of confidence intervals for ordinal coefficient alpha. Journal of Modern Applied Statistical Methods, 16(2), 157–185. https://doi.org/10.22237/jmasm/1509494940
63. Uysal, İ., & Kılıç, A. (2022). Normal dağılım ikilemi. Anadolu Journal of Educational Sciences International, 12(1), 220–248. https://doi.org/10.18039/ajesi.962653
64. van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software, 45(3). https://doi.org/10.18637/jss.v045.i03
65. Van Ginkel, J. R., Van Der Ark, L. A., & Sijtsma, K. (2007). Multiple imputation of item scores in test and questionnaire data, and influence on psychometric results. Multivariate Behavioral Research, 42(2), 387–414. https://doi.org/10.1080/00273170701360803
66. Wei, R., Wang, J., Su, M., Jia, E., Chen, S., Chen, T., & Ni, Y. (2018). Missing value imputation approach for mass spectrometry-based metabolomics data. Scientific Reports, 8(1), 663. https://doi.org/10.1038/s41598-017-19120-0
67. Zhang, Z. (2016). Missing data imputation: Focusing on single imputation. Annals of Translational Medicine, 4(1), 1–8. https://doi.org/10.3978/j.issn.2305-5839.2015.12.38
68. Zhang, Z., & Yuan, K.-H. (2016). Robust coefficients alpha and omega and confidence intervals with outlying observations and missing data: Methods and software. Educational and Psychological Measurement, 76(3), 387–411. https://doi.org/10.1177/0013164415594658