Comparison of cronbach’s alpha and McDonald’s omega for ordinal data: Are they different?

Abstract

Among all, Cronbach’s Alpha and McDonald’s Omega are commonly used for reliability estimations. The alpha uses inter-item correlations while omega is based on a factor analysis result. This study uses simulated ordinal data sets to test whether the alpha and omega produce different estimates. Their performances were compared according to the sample size, number of items, and deviance from tau equivalence. Based on the result, the alpha and omega had similar results, except for the small sample size, the smaller number of items, and the low factor loading values. When there were 5 or more items in the scale and factor analysis which the omega was calculated from showed fit to the data set, using omega over alpha could be preferred. Also, as the number of items exceeds 5, the alpha and omega differences disappear. Since calculating the alpha is easier compared to the omega (omega requires fitting a factor model first) using alpha over omega can also be suggested. However, when the number of items and the correlations among the items were small, omega performed worse than alpha. Therefore, alpha should be used for the reliability estimations.

Keywords

• Cronbach’s Alpha
• McDonald’s Omega
• Reliability
• MonteCarloSEM
• Simulation

References

1. Bandalos, D.L. (2002). The effects of item parceling on goodness-of-fit and parameter estimate bias in structural equation modeling. Structural Equation Modeling, 9(1), 78–102. https://doi.org/10.1207/S15328007SEM0901_5
2. Bernardi, R.A. (1994). Validating research results when cronbach’s alpha is below .70: A methodological procedure. Educational and Psychological Measurement, 54(3), 766–775. https://doi.org/10.1177/0013164494054003023
3. Cortina, J.M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98–104. https://doi.org/10.1037/0021-9010.78.1.98
4. Edwards, A.A., Joyner, K.J., & Schatschneider, C. (2021). A simulation study on the performance of different reliability estimation methods. Educational and Psychological Measurement, 81(6), 1089 1117. https://doi.org/10.1177/0013164421994184
5. Ercan, I., Yazici, B., Sigirli, D., Ediz, B., & Kan, I. (2007). Examining cronbach alpha, theta, omega reliability coefficients according to sample size. Journal of Modern Applied Statistical Methods, 6(1), 291-303. https://doi.org/10.22237/jmasm/1177993560
6. Gagne, P., & Hancock, G.R. (2006). Measurement model quality, sample size, and solution propriety in confirmatory factor models. Multivariate Behavioral Research, 41(1), 65-83. https://doi.org/10.1207/s15327906mbr4101_5
7. Goodboy, A.K., & Martin, M.M. (2020). Omega over alpha for reliability estimation of unidimensional communication measures. Annals of the International Communication Association, 44(4), 422-439. https://doi.org/10.1080/23808985.2020.1846135
8. Henson, R.K., Kogan, L.R., & Vacha-Haase, T. (2001). A reliability generalization study of the teacher efficacy scale and related instruments. Educational and Psychological Measurement, 61(3), 404-420. https://doi.org/10.1177/00131640121971284

9. Hu, L., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. https://doi.org/10.1080/10705519909540118
10. Kalkbrenner, M.T. (2023). Alpha, omega, and h internal consistency reliability estimates: Reviewing these options and when to use them. Counseling Outcome Research and Evaluation, 14(1), 77-88. https://doi.org/10.1080/21501378.2021.1940118
11. McDonald, R.P. (2011). Test theory: A unified treatment. Routlege.
12. Orçan, F. (2021). MonteCarloSEM: An R package to simulate data for SEM. International Journal of Assessment Tools in Education, 8(3), 704 713. https://dergipark.org.tr/en/pub/ijate/issue/62753/804203
13. R Core Team. (2014). R: A language and environment for statistical computing [Computer software manual]. http://www.R-project.org/
14. Raykov, T., & Marcoulides, G.A. (2015). A direct latent variable modeling based method for point and interval estimation of coefficient alpha. Educational and Psychological Measurement, 75(1), 146–156. https://doi.org/10.1177/0013164414526039
15. Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. https://doi.org/10.18637/jss.v048.i02
16. Signorell, A. (2023). DescTools: Tools for descriptive statistics. R package version 0.99.48. https://CRAN.R-project.org/package=DescTools
17. Streiner, D.L. (2003). Starting at the Beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80(1), 99 103. https://doi.org/10.1207/S15327752JPA8001_18
18. Vaske, J.J., Beaman, J., & Sponarski, C.C. (2017). Rethinking internal consistency in cronbach's alpha. Leisure Sciences, 39(2), 163 173. https://doi.org/10.1080/01490400.2015.1127189
19. Viladrich, C., Angulo-Brunet, A., & Doval, E. (2017). A journey around alpha and omega to estimate internal consistency reliability. Anales de Psicología / Annals of Psychology, 33(3), 755–782. https://doi.org/10.6018/analesps.33.3.268401
20. Yurdagül, H., (2008). Minimum sample size for cronbach’s coefficient alpha: A Monte-Carlo Study. H.U. Journal of Education, 35, 397 405. http://www.efdergi.hacettepe.edu.tr/shw_artcl-571.html