Measurement Invariance Testing with Alignment Method: Many Groups Comparison

Abstract

It is necessary to examine the measurement invariance (MI) among groups in studies where different groups are compared by using a measurement instrument. Most of the studies, measurement invariance is tested with multiple group confirmatory factor analysis. This model applies many model adjustments based on the modification indexes. Therefore, it is not practical due to too many large modification indexes while testing MI over many groups. Besides scalar model is a poor model fit when comparing many groups and so does not hold MI. In this study, the aim is to explain the basic concepts and processes of the alignment method which is offered as a new method for testing MI and illustrate an application on the real data set. In this study, measurement invariance among 56 countries including Turkey is tested with alignment method in order to set an example for researchers. For this purpose, the Instrumental Motivation Scale data, which is one of the psychological measurement instruments used in PISA 2015, was used. As a result of MG-CFA, it was found that configural invariance was ensured. The fit indexes of CFI and TLI were calculated as 0.982 and 0.946 respectively in this stage. After that, metric invariance was tested by considering the difference of fit indices obtained for the two stages. It was found that the metric invariance could not be provided. Alignment results show which countries hold MI and which do not. Besides it provides information which items have the most invariants for groups that hold MI.

Keywords

• multi-group confirmatory factor analysis
• validity
• modification indexes

References

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2. Asil, M., & Gelbal, S. (2012). PISA öğrenci anketinin kültürler arası eşdeğerliği. Eğitim ve Bilim, 37(166), 236-249.
3. Asil, M., & Brown, G. T. L. (2015). Comparing OECD PISA reading in English to other languages: Identifying potential sources of non-invariance. International Journal of Testing, (16)1, 71-93. https://doi.org/10.1080/15305058.2015.1064431
4. Asparouhov, T., & Muthén, B. (2014). Multiple-group factor analysis alignment. Structural Equation Modeling: A Multidisciplinary Journal, 21(4), 495 508. https://doi.org/10.1080/10705511.2014.919210
5. Byrne, B.M., Shavelson, R.J., & Muthén, B.O. (1989). Testing for equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105(3), 456-466.
6. Camilli, G., & Shepard, L. A. (1994). Methods for identifying biased test items (4th ed.). Thousand Oaks, California, USA: Sage Publications.
7. Drasgow, F., & Kanfer, R. (1985). Equivalence of psychological measurement in heterogeneous populations. Journal of Applied Psychology, 70(4), 662-80.
8. Ercikan, K., & Koh, K. (2005). Examining the construct comparability of the English and French versionf of TIMSS. International Journal of Testing, 5(1), 23-35. https://doi.org/10.1207/s15327574ijt0501_3

9. Fan, X., & Sivo, S. A. (2009). Using Δ-goodness-of-fit indexes in assessing mean structure invariance. Structural Equation Modeling: A Multidisciplinary Journal, 16(1), 54-69, https://doi.org/10.1080/10705510802561311
10. Gülleroğlu, H. D. (2017). PISA 2012 matematik uygulamasına katılan türk öğrencilerin duyuşsal özelliklerinin cinsiyete göre ölçme değişmezliğinin incelenmesi, Gazi Üniversitesi Gazi Eğitim Fakültesi Dergisi, 37(1), 151-175.
11. Gür, E. (2019). PISA 2015 uygulamasındaki maddelerin kültüre göre değişen madde fonksiyonu acısından incelenmesi (Master's thesis). Hacettepe University, Graduate School of Educational Sciences, Ankara, Türkiye.
12. Hansson, Å., & Gustafsson, J. (2013). Measurement invariance of socioeconomic status across migrational background, Scandinavian Journal of Educational Research, 57(2), 148-166. https://doi.org/10.1080/00313831.2011.625570
13. İmrol, F. (2017). Pisa 2012 Türkiye örnekleminde matematiğe yönelik motivasyon ve öz-inanç yapılarının ölçme değişmezliğinin incelenmesi (Master's thesis). Ankara University Graduate School of Educational Sciences, Ankara, Türkiye.
14. Jennrich, R. I. (2006). Rotation to simple loadings using component loss functions: The oblique case. Psychometrika, 71(1), 173-191. https://doi.org/10.1007/s11336-003-1136-B
15. Jöreskog, K.G., Sörbom, D., Du Toit, S.H.C., & Du Toit, M. (2001). LISREL 8: New statistical features (3rd ed.). Lincolnwood, IL: Scientific Software International.
16. Kıbrıslıoğlu, N. (2015). PISA 2012 matematik öğrenme modelinin kültürlere ve cinsiyete göre ölçme değişmezliğinin incelenmesi: Türkiye, Çin (Şangay)-Endonezya örneği. (Master’s thesis, Hacettepe University, Graduate School of Educational Sciences, Ankara, Türkiye). Retrieved from http://openaccess.hacettepe.edu.tr:8080/xmlui/bitstream/handle/11655/1843/70bd9399-b70f-414a-a11f-b81218adb77c.pdf?sequence=1
17. Kim, E. S., Cao, C., Wang, Y., & Nguyen, D. T. (2017). Measurement invariance testing with many groups: a comparison of five approaches, Structural Equation Modeling: A Multidisciplinary Journal, 24(4), 524 544. https://doi.org/10.1080/10705511.2017.1304822
18. Luo, C. (2010). Measurement invariance of Rosenberg Self-Esteem Scale between British and Chinese college students. (Master's thesis The University of Edinburgh, Edinburgh, Scotland). Retrieved from https://pdfs.semanticscholar.org/274b/015fedaa0c835d94680df2a3c153aff36e5a.pdf
19. Marsh, H. W., Guo, J., Parker, P. D., Nagengast, B., Asparouhov, T., Muthén, B., & Dicke, T. (2018). What to do when scalar invariance fails: The extended alignment method for multi-group factor analysis comparison of latent means across many groups. Psychological Methods, 23(3), 524-545. https://doi.org/10.1037/met0000113
20. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. ETS Research Report Series, 2, i-28. https://doi.org/10.1002/j.2333-8504.1994.tb01618.x
21. Muthén, B., & Asparouhov, T. (2018). Recent methods for the study of measurement invariance with many groups: alignment and random effects. Sociological Methods & Research, 47(4) 637-664. https://doi.org/10.1177/0049124117701488
22. Pauwels, J. (2018) Do you really measure the same? (Doctoral dissertation, Ghent University, Gent, Belgium). Retrieved from: https://lib.ugent.be/fulltxt/RUG01/002/481/799/RUG01-002481799_2018_0001_AC.pdf
23. OECD (2016). PISA 2015 Results (Volume II): Policies and Practices for Successful Schools, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264267510-en
24. Rutkowski, L., & Svetina, D. (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74(1), 31–57. https://doi.org/10.1177/ 0013164413498257
25. Sırgancı, G., & Çakan, M. (2020). Sirali lojistik regresyon ve poly-sibtest yöntemleri ile değişen madde fonksiyonunun belirlenmesi. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 20(1), 705-717.
26. Sokolov, B. (2019). Sensitivity of goodness of fit indices to lack of measurement invariance with categorical indicators and many groups. Higher School of Economics Research Paper No. WP BRP 86/SOC/2019. Retrieved from https://wp.hse.ru/data/2019/07/09/1480015921/86SOC2019.pdf
27. Sörbom, D. (1989). Model modification. Psychometrika, 54, 371–384.
28. Tiryaki, F. (2019). PISA 2015 öğrenci tutum anketlerinin değişen madde fonksiyonu ve ölçme değişmezliğinin incelenmesi (Master’s Thesis). Ankara University Graduate School of Educational Sciences, Ankara, Türkiye
29. Uyar, Ş. (2011). PISA 2009 Türkiye örnekleminde öğrenme stratejileri modelinin farklı gruplarda ölçme değişmezliğinin incelenmesi (Master’s Thesis). Hacettepe University Graduate School of Educational Sciences, Ankara, Türkiye
30. Uyar, Ş., & Uyanık, G. K. (2019). Fen bilimlerine yönelik öğrenme modelinin ölçme değişmezliğinin incelenmesi: Pisa 2015 örneği. Kastamonu Eğitim Dergisi, 27(2), 497-507. https://doi.org/10.24106/kefdergi.2570
31. Uzun, B., & Öğretmen, T. (2010). Fen başarısı ile ilgili bazı değişkenlerin timss-r türkiye örnekleminde cinsiyete göre ölçme değişmezliğinin değerlendirilmesi. Eğitim ve Bilim, 35(155), 26-35.
32. Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the MI literature: suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-70. https://doi.org/10.1177/109442810031002
33. Widaman, K. F., & Reise, S. P. (1997). Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. In K. J. Bryant, M. Windle, & S. G. West (Eds.), The science of prevention: Methodological advances from alcohol and substance abuse research (p. 281–324). Washington DC, USA: American Psychological Association. https://doi.org/10.1037/10222-009
34. Wu, A. D., Li, Z., & Zumbo, B. D. (2007). Decoding the meaning of factorial ınvariance and updating the practice of multigroup confirmatory factor analysis: A demonstration with TIMSS data. Practical Assessment, Research and Evaluation, 12, 1-26. https://doi.org/10.7275/mhqa-cd89
35. Yandı, A., Köse, İ. A., & Uysal, Ö. (2017). Farklı yöntemlerle ölçme değişmezliğinin incelenmesi: PISA 2012 örneği. Mersin Üniversitesi Eğitim Fakültesi Dergisi, 13(1), 243-253. https://doi.org/10.17860/mersinefd.305952
36. Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and likert-type (Ordinal) item scores. Ottawa ON: Directorate of Human Resources Research and Evaluation, Department of National Defense.