Cross-National Measurement of Mathematics Intrinsic Motivation: An Investigate of Measurement Invariance with MG-CFA and Aligment Method Across Fourteen Countries

Abstract

One of the main objectives of international large-scale assessments is to make comparisons between different countries, education policies, education systems, or subgroups. One of the main criteria for making comparisons between different groups is to ensure measurement invariance. The purpose of this study was to test the measurement invariance of the mathematics intrinsic motivation scale across 14 countries. For this purpose, the "students like learning mathematics" scale, which measures intrinsic motivation for mathematics, was included in the TIMSS 2019 cycle. The study sample consisted of a total of 152992 students, 70192 4th grade and 82800 8th grade students from 14 different countries participating in the TIMSS 2019 cycle. Measurement invariance was tested with Multi-Group Confirmatory Factor Analysis (MG-CFA) and Alignment Method. The mathematics intrinsic motivation scale provides only configural invariance according to MG-CFA at the 4th grade level, whereas the scale provides approximate invariance according to the alignment method. At the 8th grade level, the scale provides configural and metric invariance according to MG-CFA, whereas the scale provides approximate invariance according to the alignment method. The results indicate that the mathematics intrinsic motivation scale provides approximate measurement invariance at both grade levels and that comparisons can be made between the scores of the identified countries.

Keywords

• mathematics
• motivation
• measurement invariance
• students like learning mathematics
• alignment method

Matematikte İçsel Motivasyonun Ülkeler Arası Ölçümü: On Dört Ülkede MG-CFA ve Hizalama Yöntemi ile Ölçme Değişmezliğinin İncelenmesi

Öz

Geniş ölçekli uluslararası değerlendirmelerin temel amaçlarından biri, farklı ülkeler, eğitim politikaları, eğitim sistemleri veya alt gruplar arasında karşılaştırmalar yapmaktır. Farklı gruplar arasında karşılaştırma yapmanın temel ölçütlerinden biri de ölçme değişmezliğinin sağlanmasıdır. Bu çalışmanın amacı, matematik içsel motivasyon ölçeğinin 14 ülke arasında ölçme değişmezliğini test etmektir. Bu amaçla, matematiğe yönelik içsel motivasyonu ölçen "öğrenciler matematik öğrenmeyi sever" ölçeği TIMSS 2019 döngüsüne dahil edilmiştir. Çalışmanın örneklemi TIMSS 2019 döngüsüne katılan 14 farklı ülkeden 70192 4. sınıf ve 82800 8. sınıf öğrencisi olmak üzere toplam 152992 öğrenciden oluşmaktadır. Ölçme değişmezliği, Çok Gruplu Doğrulayıcı Faktör Analizi (MG-CFA) ve Hizalama Yöntemi ile test edilmiştir. Matematik içsel motivasyon ölçeği, 4. sınıf düzeyinde MG-CFA'ya göre sadece yapısal değişmezliği sağlarken, hizalama yöntemine göre yaklaşık değişmezliği sağlamaktadır. 8. sınıf düzeyinde ise ölçek, MG-CFA'ya göre konfigüral ve metrik değişmezliği sağlarken, hizalama yöntemine göre yaklaşık değişmezliği sağlamaktadır. Sonuçlar, matematik içsel motivasyon ölçeğinin her iki sınıf düzeyinde de yaklaşık ölçme değişmezliğini sağladığını ve belirlenen ülkelerin puanları arasında karşılaştırmalar yapılabileceğini göstermektedir.

Anahtar Kelimeler

• matematik
• motivasyon
• ölçme değişmezliği
• hizalama yöntemi
• öğrenciler matematik öğrenmeyi sever

References

1. Adıbatmaz, F. B. K., & Yildiz, H. (2020). The Effects of Distractors to Differential Item Functioning in Peabody Picture Vocabulary Test. Journal of Theoretical Educational Science, 13(3), 530-547.
2. Afanador, N. L., Tran, T., Blanchet, L., & Baumgartner, R. (2016). mvdalab-package 3.
3. Ahmed, W., Minnaert, A., Van der Werf, G., & Kuyper, H. (2010). Perceived social support and early adolescents' achievement: The mediational roles of motivational beliefs and emotions. Journal of Youth and Adolescence, 39(1), 36–46. doi:10.1007/s10964-008-9367-7
4. Akben-Selcuk, E. (2017). Personality, motivation, and math achievement among Turkish students: Evidence from PISA data. Perceptual and Motor Skills, 124(2), 514–530. https://doi.org/10.1177/0031512516686505
5. Arikan, S., Özer, F., Şeker, V., & Ertaş, G. (2020). The importance of sample weights and plausible values in large-scale assessments. Journal of Measurement and Evaluation in Education and Psychology, 11(1), 43-60. doi: https://doi.org/10.21031/epod.602765
6. Barak, M., & Asad, K. (2012). Teaching image-processing concepts in junior high schools: Boys' and girls' achievements and attitudes towards technology. Research in Science and Technological Education, 30(1), 81–105. doi:10.1080/02635143. 2012. 656084
7. Başusta, N. B., & Gelbal, S. (2015). Examination of Measurement Invariance at Groups' Comparisons: A Study on PISA Student Questionnaire. Hacettepe University Education Faculty Journal, 30(4), 80-90.
8. Büyüköztürk, Ş., Çakmak, E. K., Akgün, Ö. E., Karadeniz, Ş., & Demirel, F. (2017). Scientific research methods. Pegem, 1-360.

9. Cardoso, M. E. (2020). Policy evidence by design: International large-scale assessments and grade repetition. Comparative Education Review, 64(4), 598-618. doi: https://doi.org/10.1086/710777
10. Cheung G. W., Rensvold R. B. (1999). Testing factorial invariance across groups: A reconceptualization and proposed new method. Journal of Management, 25(1), 1-27. https://doi.org/10.1177/014920639902500101
11. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural equation modeling, 9(2), 233-255.
12. Cleary, T. J., & Chen, P. P. (2009). Self-regulation, motivation, and math achievement in middle school: Variations across grade level and math context. Journal of School Psychology, 47(5), 291–314. doi: 10.1016/j.jsp.2009.04.002
13. Engel, L. C., & Rutkowski, D. (2021). Costs of big data. Digital Disruption In Teaching And Testing (pp. 124–135). Routledge.
14. Ertürk, Z., & Erdinç-Akan, O. (2018). TIMSS 2015 matematik başarısı ile ilgili bazı değişkenlerin cinsiyete göre ölçme değişmezliğinin incelenmesi. Journal of Theoretical Educational Science, 204-226.
15. Fischer, R., & Karl, J. A. (2019). A primer to (cross-cultural) multi-group invariance testing possibilities in R. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.01507
16. Forero, C. G., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 625–641. https://doi.org/10.1080/10705510903203573
17. Freiberger, V., Steinmayr, R., & Spinath, B. (2012). Competence beliefs and perceived ability evaluations: How do they contribute to intrinsic motivation and achievement?. Learning and individual differences, 22(4), 518-522. doi: https://doi.org/10.1016/j.lindif.2012.02.004
18. Glassow, L. N., Rolfe, V., & Hansen, K. Y. (2021). Assessing the comparability of teacher-related constructs in TIMSS 2015 across 46 education systems: an alignment optimization approach. Educational Assessment Evaluation and Accountability, 33(1), 105–137. https://doi.org/10.1007/s11092-020-09348-2
19. Gregorich, S. E. (2006). Do self-report instruments allow meaningful comparisons across diverse population groups? Testing measurement invariance using the confirmatory factor analysis framework. Medical care, 44(11 Suppl 3), S78.
20. Guo, J., Parker, P. D., Marsh, H. W., & Morin, A. J. S. (2015). Achievement, motivation, and educational choices: A longitudinal study of expectancy and value using a multiplicative perspective. Developmental Psychology, 51(8), 1163–1176. https://doi.org/10.1037/a0039440
21. Gustafsson, J.-E. (2018). International large scale assessments: Current status and ways forward. Scandinavian Journal of Educational Research, 62(3), 328–332. https://doi.org/10.1080/00313831.2018.1443573
22. He, J., Barrera-Pedemonte, F., & Buchholz, J. (2019). Cross-cultural comparability of noncognitive constructs in TIMSS and PISA. Assessment in Education Principles Policy and Practice, 26(4), 369–385. https://doi.org/10.1080/0969594x.2018.1469467
23. Henderson, R. W. & Landesman, E. M. (1995). Effects of thematically integrated mathematics instruction on students of Mexican descent. Journal of Educational Research, 88(5), 290–300.
24. Hooper, M., Mullis, I. V., Martin, M. O., & Fishbein, B. (2020). TIMSS 2019 context questionnaire framework. TIMSS, 59-78.
25. Horn, J. L., & McArdle, J. J. (1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18(3–4), 117–144. https://doi.org/10.1080/03610739208253916
26. Hu, L.-T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
27. Ilter, İ. (2021). The relationship between academic amotivation and academic achievement: A study on middle school students. Kuramsal Eğitimbilim Dergisi, 14(3), 389–410. https://doi.org/10.30831/akukeg.847145
28. İlhan, M., & Çetin, B. (2013). Matematik odaklı epistemolojik inanç ölçeği (MOEİÖ): Geçerlik ve güvenirlik çalışması. Kurumsal Eğitimbilim Dergisi, 362-368.
29. Jami, W. A., & Kemmelmeier, M. (2020). Assessing well-being across space and time: Measurement equivalence of the WHO-5 in 36 European countries and over 8 years. Journal of Well-Being Assessment, 4(3), 419–445. https://doi.org/10.1007/s41543-021-00042-8
30. Jöreskog, K. G., & Sörbom, D. (1993). LISREL 8: Structural equation modeling with the SIMPLIS command language. Scientific software international.
31. Kaliyaperumal, S. K., Kuppusamy, M., & Gounder, A. S. (2015). Outlier detection and missing value in time series ozone data. International Journal of Scientific Research in Knowledge, 3(9), 220–226. https://doi.org/10.12983/ijsrk-2015-p0220-0226
32. Kam, C. C. S. (2019). Careless responding threatens factorial analytic results and construct validity of personality measure. Frontiers in Psychology, 10, 1258. https://doi.org/10.3389/fpsyg.2019.01258
33. Kam, C. C. S., & Meyer, J. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18(3), 512–541. https://doi.org/10.1177/1094428115571894
34. Kaya, S., Eryilmaz, N., & Yuksel, D. (2023). A cross-cultural comparison of self-efficacy as a resilience measure: Evidence from PISA 2018. Youth & Society. https://doi.org/10.1177/0044118x231186833
35. Kline, R. B. 2011. "Convergence of Structural Equation Modeling and Multilevel Modeling." In The SAGE Handbook of Innovation in Social Research Methods, edited by M. Williams and W. P. Vogt, 562–589. SAGE Publications. doi:10.4135/9781446268261.
36. Koğar, H., & Yilmaz Koğar, E. (2015). Comparison of different estimation methods for categorical and ordinal data in confirmatory factor analysis. Journal of Measurement and Evaluation in Education and Psychology, 6(2). https://doi.org/10.21031/epod.94857
37. Malone, T. W., & Lepper, M. R. (2021). Making learning fun: A taxonomy of intrinsic motivations for learning. In Aptitude, learning, and instruction (pp. 223-254). Routledge.
38. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525–543.
39. Middleton, J. (1995). A study of intrinsic motivation in the mathematics classroom: A personal constructs approach. Journal for Research in Mathematics Education, 26(3), 254–279. doi:10.2307/749130
40. Milfont, T. L., & Fischer, R. (2010). Testing measurement invariance across groups: Applications in cross-cultural research. International Journal of psychological research, 3(1), 111-130.
41. Millsap, R. E., & Olivera-Aguilar, M. (2012). Investigating measurement invariance using confirmatory factor analysis. In R. H. Hoyle, (Ed.) Handbook of structural equation modeling, (pp. 380-392), Guilford.
42. Mueller, M., Yankelewitz, D., & Maher, C. (2011). Sense making as motivation in doing mathematics: Results from two studies. The Mathematics Educator, 20(2), 33–43.
43. Mullis, I. V. S., & Martin, M. O. (Eds.). (2017). TIMSS 2019 Assessment Frameworks. Retrieved from Boston College, TIMSS and PIRLS International Study Center website: http://timssandpirls.bc.edu/timss2019/frameworks/
44. Muthén ©n, B., & Asparouhov, T. (2014). IRT studies of many groups: the alignment method. Frontiers in Psychology, 5. https://doi.org/10.3389/fpsyg.2014.00978
45. Nugent, G., Barker, B., Grandgenett, N., & Adamchuk, V. (2010). Impact of robotics and geospatial technology interventions on youth stem learning and attitudes. Journal of Research on Technology in Education, 42(4), 391–408.
46. OECD. (2013). Students' drive and motivation. In PISA 2012 results: Ready to learn (Volume III): Students' engagement, drive, and self-beliefs. OECD Publishing. Retrieved from http://dx.doi.org/10.1787/ 9789264201170-7-en.
47. Peterson, B. G., Carl, P., Boudt, K., Bennett, R., Ulrich, J., Zivot, E., ... & Wuertz, D. (2018). Package 'performanceanalytics'. R Team Cooperation, 3, 13-14.
48. Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review: DR, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004
49. Raižienė, S., Ringienė, L., Laukaityte, I., & Jakaitienė, A. (2021). Measurement invariance of pisa 2018 motivational constructs across eu countries. EDULEARN21 Proceedings (pp. 7081-7081). IATED.
50. Raykov, T. (2004). Behavioral scale reliability and measurement invariance evaluation using latent variable modeling. Behavior Therapy, 35(2), 299–331. https://doi.org/10.1016/s0005-7894(04)80041-8
51. Robitzsch, A. (2020). sirt: Supplementary item response theory models. R package version 3.4-64. https://CRAN.R-project.org/package=sirt
52. Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2). https://doi.org/10.18637/jss.v048.i02
53. Rosseel, Y. (2012). lavaan: AnRPackage for Structural Equation Modeling. Journal of Statistical Software, 48(2). https://doi.org/10.18637/jss.v048.i02
54. Rutkowski, L., & Svetina, D. (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74(1), 31–57. https://doi.org/10.1177/0013164413498257
55. Ryan, R.M., & Deci E.L., (2009) Promoting self-determined school engagement: motivation, learning, and well-being. In: Wentzel KR, Wigfield A (eds) Handbook on motivation at school. Routledge, New York, pp 171–196.
56. Schmitt, N., & Kuljanin, G. (2008). Measurement invariance: Review of practice and implications. Human Resource Management Review, 18(4), 210–222. https://doi.org/10.1016/j.hrmr.2008.03.003
57. Shores, M. L., & Shannon, D. M. (2007). The effects of self-regulation, motivation, anxiety, and attributions on mathematics achievement for fifth and sixth grade students. School Science and Mathematics, 107(6), 225–236. Retrieved from http://ssmj.tamu.edu
58. Sırgancı, G., Uyumaz, G., & Yandi, A. (2020). Measurement invariance testing with alignment method: Many groups comparison. International Journal of Assessment Tools in Education, 7(4), 657–673. https://doi.org/10.21449/ijate.714218
59. Sözer, E., Eren, B., & Kahraman, N. (2021). Investigating measurement invariance for longitudinal assessments: An application using repeated data over four weeks. Gazi Üniversitesi Gazi Eğitim Fakültesi Dergisi, 41(2), 729–763. https://doi.org/10.17152/gefad.873885
60. Tabachnick, B. G. & Fidell, L. S. (2013). Using multivariate statistics (6th edition). Northridge: Pearson.
61. Taris, T. W., Bok, I. A., & Meijer, Z. Y. (1998). Assessing stability and change of psychometric properties of multi-item concepts across different situations: A general approach. The Journal of Psychology, 132(3), 301–316. https://doi.org/10.1080/00223989809599169
62. Tavani C.M., & Losh S.C. (2003) Motivation, self-confidence, and expectations as predictors of the academic performances among our high school students. Child Study J 33(3):141–151.
63. Tierney, N., Cook, D., McBain, M., & Fay, C. (2021). naniar: Data structures, summaries, and visualisations for missing data (R package version 0.6. 1)[Computer software].
64. Uyar, Ş. & Doğan, N. (2014). An Investigation Of Measurement Invariance Of Learning Strategies Model Across Different Groups in Pisa Turkey Sample. International Journal Of Turkish Education Sciences, 2014(3), 30-43.
65. Van De Schoot, R., Schmidt, P., De Beuckelaer, A., Lek, K., & Zondervan-Zwijnenburg, & M. (2015). Editorial: Measurement invariance. Frontiers in Psychology, 6, 1064. https://doi.org/10.3389/fpsyg.2015.01064
66. Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70. https://doi.org/10.1177/109442810031002
67. Weidinger, A. F., Steinmayr, R., & Spinath, B. (2017). Math grades and intrinsic motivation in elementary school: A longitudinal investigation of their association. British Journal of Educational Psychology, 87(2), 187-204.
68. Wickham, H., François, R., Henry, L., Müller, K., & Wickham, M. H. (2019). Package 'dplyr'. A Grammar of Data Manipulation. R package version, 8.
69. Woods, C. M. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28(3), 186-191.
70. Woolley, M. E., Strutchens, M. E., Gilbert, M. C., & Martin, W. (2010). Mathematics success of black middle school students: Direct and indirect effects of teacher expectations and reform practices. Negro Educational Review, 61(1), 41–59. Retrieved from http://oma. osu. edu/vice_provost/ner/index. Html
71. Wu, A. D., Li, Z., & Zumbo, B. D. (2007). Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analysis: A demonstration with TIMSS data. University of Massachusetts Amherst. https://doi.org/10.7275/MHQA-CD89
72. Yi̇ği̇ter, M. S. (2023). Matematik Duyuşsal Özellik Faktörlerinin Cinsiyete Göre Ölçme Değişmezliğinin İncelenmesi: TIMSS 2019 Türkiye Örneği. Anadolu Üniversitesi Eğitim Fakültesi Dergisi, 7(4), 859–882. https://doi.org/10.34056/aujef.1198134
73. Yildirim, S. (2011). Self-efficacy, intrinsic motivation, anxiety and mathematics achievement: Findings from Turkey, Japan and Finland. Necatibey Faculty of Education Electronic Journal of Science and Mathematics Education, 5(1), 277-291.
74. Yin, L., & Fishbein, B. (2019). Creating and interpreting the TIMSS 2019 context questionnaire scales. Methods and procedures: TIMSS, 16-1.
75. Zembat, R., Akşin-Yavuz, E., Tunçeli, H. İ., Yılmaz, H. (2018). Öğretmenlik mesleğine yönelik tutum ile akademik motivasyon ve başarı arasındaki ilişkinin incelenmesi. Kuramsal Eğitimbilim Dergisi [Journal of Theoretical Educational Science], 11(4), 789-808.