Comparison of Alpha, Stratified Alpha, and Omega Reliability Coefficients in Multidimensional Test Structures

Abstract

The aim of this research is to examine the reliability of multidimensional test structures. For this purpose, the estimations of the Cronbach Alpha, Stratified Alpha and Omega coefficients for a two-dimensional test were examined in the the data simulated under sample size (100,200 and 400), correlation between dimensions (0.00 and 0.50), correlation between items (0.30-0.50 and ≥0.65), test length (10 and 20) and the number of items per dimensions (5-5, 7-3, 10-10 and 15-1). As a result of the research, it was found that the Stratified Alpha and Omega values were higher than the general Alpha value.. In addition, it was concluded that the inter-item correlation condition was more effective than the interdimensional correlation on the estimations of the three reliability coefficients.

Keywords

• reliability
• Cronbach alpha
• stratified alpha
• omega

Çok Boyutlu Test Yapılarında Alfa, Tabakalı Alfa ve Omega Güvenirlik Katsayılarının Karşılaştırılması

Öz

Bu araştırmanın amacı, çok boyutlu test yapılarının güvenirliğinin incelenmesidir. Bu amaçla iki boyutlu bir test için örneklem büyüklüğü (100,200 ve 400), boyutlar arası korelasyon (0,00 ve 0,50), maddeler arası korelasyon (0,30-0,50 ve ≥0,65) , test uzunluğu (10 ve 20) ve boyutlara düşen madde sayısı (5-5, 7-3, 10-10 ve 15-1) koşulları altında üretilen verilerden Cronbach Alfa, Tabakalı Alfa ve Omega katsayılarının kestirim değerleri incelenmiştir. Araştırma sonucunda Tabakalı Alfa ve Omega değerlerinin genel Alfa değerinden yüksek fakat birbirine benzer sonuçlar verdiği bulunmuştur. Ayrıca üç güvenirlik katsayısının kestirimleri üzerinde maddeler arası korelasyon koşulunun boyutlar arası korelasyona göre daha etkili olduğu sonucuna ulaşılmıştır.

Anahtar Kelimeler

• güvenirlik
• Cronbach alfa
• tabakalı alfa
• omega

References

1. Bacon, D. R., Sauer, P. L., & Young, M. (1995). Composite Reliability in Structural Equations Modeling. Educational and Psychological Measurement, 55(3), 394-406.
2. Brunner, M., & Süβ, H. M. (2005). Analyzing the reliability of multidimensional measures: An example from intelligence research. Educational and Psychological Measurement, 65(2), 227-240.
3. Carmines, E. G., & Zeller, R. A. (1982). Reliability and validity assessment. Beverly Hills: Sage Publications.
4. Cho, E., & Kim, S. (2015). Cronbach’s coefficient alpha: Well-known but poorly understood. Organizational Research Methods, 18, 207-230.
5. Cho, E. (2016). Making reliability reliable: A systematic approach to reliability coefficients. Organizational Research Methods, 19(4), 651-682.
6. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart & Winston.
7. Cronbach, L. J. (1947). Test “reliability”: Its meaning and determination. Psychometrika, 12(1), 1–16.
8. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.

9. Cronbach, L. J., Schonemann, P., & McKie, D. (1965). Alpha coefficients for stratified-paralel tests. Educational and Psychological Measurement, 25, 291-312.
10. Cortina, J. M (1993). What is coefficient alpha? An examination of theory and application. Journal of Applied Psychology, 78, 98-104.
11. Deng, L., & Chan, W. (2017). Testing the difference between reliability coefficients alpha and omega. Educational and psychological measurement, 77(2), 185-203.
12. Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British journal of psychology, 105(3), 399-412.
13. Feldt, L. S. , Brennan, R. L. (1989) Reliability. In Linn RL (ed), Educational measurement (3rd ed., pp. 105–146), Macmillan.
14. Graham, J. M. (2006). Congeneric and (Essentially) Tau-Equivalent Estimates of Score Reliability. Educational and Psychological Measurement, 66(6), 930–944.
15. Green, S. B., Lissitz, R.W., and Mulaik, S. A. (1977). Limitations of coefficient alpha as an index of test unidimensionality. Educational and Psychological Measurement, 37, 827–838.
16. Guttman, L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10, 255-282.
17. Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis. Pearson College Division.
18. Harwell, M., Stone, C. A., Hsu, T. C., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20(2), 101-125.
19. Hayes, A. F., & Coutts, J. J. (2020). Use omega rather than Cronbach’s alpha for estimating reliability. But…. Communication Methods and Measures, 14(1), 1-24.
20. Hoyt, C. (1941). Test reliability estimated by analysis of variance. Psychometrika, 6(3), 153–160
21. Kamata, A., Turhan, A., & Darandari, E. (2003, April). Estimating reliability for multidimensional composite scale scores. In annual meeting of American Educational Research Association, Chicago, IL.
22. Kaplan, R. M., & Saccuzzo, D. P. (2009). Psychological testing: Principles, applications, and issues (7. basım). CA: Thompson Wadsworth.
23. McDonald, R. P. (1999). Test theory: Unified treatment. Lawrence Erlbaum Associates.
24. McDonald, R. P. (1970). Theoretical foundations of principal factor analysis and alpha factor analysis. British Journal of Mathematical and Statistical Psychology, 23, 1-21
25. Nunnally, J. C. (1978). Psychometric theory (2. Basım). New York : McGraw-Hill.
26. Osburn, H. G. (2000). Coefficient alpha and related internal consistency reliability coefficients. Psychological Methods, 5, 343-355.
27. Paek, I., and Cole, I. (2019). Using R for Item Response Theory Applications. New York: Routledge.
28. Raykov, T. (1998). Coefficient alpha and composite reliability with interrelated nonhomogeneous items. Applied Psychological Measurement, 22, 375-385.
29. Raykov, T., West, B. T., & Traynor, A. (2015). Evaluation of coefficient alpha for multiple-component measuring instruments in complex sample designs. Structural Equation Modeling: A Multidisciplinary Journal, 22(3), 429-438.
30. Revelle, W. (1979). Hierarchical cluster analysis and the internal structure of tests. Multivariate Behavioral Research,.14, 57–74.
31. Revelle, W., & Zinbarg, R. E. (2009). Coefficients alpha, beta, Omega, and the glb: Comments on Sijtsma. Psychometrika, 74, 145-154.
32. Revelle, W., & Revelle, M. W. (2022). psych: Procedures for Psychological, Psychometric, and Personality Research. Version R package version 2.2.9. https://cran.r-project.org/package=psych
33. Robitzsch, A. (2022). sirt:Supplementary Item Response Theory Models. Version R package version 3.12-66. https://cran.r-project.org/package=sirt
34. Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350–353.
35. Serbetar, I., & Sedlar, I. (2016). Assessing reliability of a multi-dimensional scale by coefficient alpha. Revija za Elementarno Izobrazevanje, 9(1/2), 189.
36. Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107-120.
37. Sijtsma, K., & Pfadt, J. M. (2021). Part II: On the use, the misuse, and the very limited usefulness of Cronbach’s alpha: Discussing lower bounds and correlated errors. Psychometrika, 86(4), 843-860.
38. Soĉan, G. (2000). Assessment of reliability when test items are not essentially tau-equivalent. Advances in Methodology and Statistics, 15, 23–35.
39. Tang, W., & Cui, Y. (2012, April). A simulation study for comparing three lower bounds to reliability. Paper presented at the annual meeting of the American Educational Research Association, Vancouver, Canada
40. Traub, E. R. (1994). Reliability for the social sciences: Theory and Applications. Measurement methods for the social sciences. London: Sage Publications.
41. Viladrich, C., Angulo-Brunet, A., & Doval, E. (2017). A journey around alpha and Omega to estimate internal consistency reliability. Anales de Psicología/Annals of Psychology, 33(3), 755-782.
42. Warrens, M. J. (2016). A comparison of reliability coefficients for psychometric tests that consist of two parts. Advances in Data Analysis and Classification, 10(1), 71-84. https://10.1007/s11634-015-0198-6
43. Watkins, M. W. (2017). The reliability of multidimensional neuropsychological measures: from alpha to Omega. The Clinical Neuropsychologist, 31(6-7), 1113-1126.
44. Wells, C. S., & Wollack, J. A. (2003). An instructor’s guide to understanding test reliability. Testing & Evaluation Services University of Wisconsin.
45. Widhiarso, W., & Ravand, H. (2014). Estimating reliability coefficient for multidimensional measures: A pedagogical illustration. Review of Psychology, 21(2), 111-121.
46. Zhang, J. (2007). Conditional covariance theory and DETECT for polytomous items. Psychometrika 72:69.
47. Zinbarg, R., Revelle, W., Yovel, I., & Li, W. (2005). Cronbach’s , Revelle’s , and Mcdonald’s : Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70, 123-133.
48. Zinbarg, R., Yovel, I., Revelle, W., & McDonald, R. (2006). Estimating generalizability to a latent variable common to all of a scale’s indicators: A comparison of estimators for estimators for ωh. Applied Psychological Measurement, 30, 121-144
49. Zinbarg, R. E., Revelle, W., & Yovel, I. (2007). Estimating for structures containing two group factors: Perils and prospects. Applied Psychological Measurement, 31, 135-157.