Investigation of Measurement Invariance of Mathematics Affective Characteristic Factors According to Gender: TIMSS 2019 Turkey Sample

Abstract

One of the main objectives of large-scale assessments is to draw conclusions about education policies or education systems by making comparisons between different countries or subgroups. One of the main criteria for making comparisons between different groups is to satisfy measurement invariance. Measurement invariance indicates that the measured construct is psychometrically equivalent between groups. Claims of differences in comparisons without evidence of measurement invariance can be unreliable. The aim of this study was to test the measurement invariance of the model created with mathematics affective characteristics according to gender. For this purpose, the Mathematics Affective Characteristics Model was created with the scales of Like Learning Mathematics (MOS), Instructional Clarity in Mathematics Lessons (MON), Disorderly Behavior During Mathematics Lessons (MDDD), Students Confident in Mathematics (MKG) and Students Value Mathematics (MDV) in the TIMSS 2019 cycle. The sample of the study consists of 3658 students from Turkey who participated in the TIMSS 2019 cycle at the 8th grade level. In the first part of the study, Confirmatory Factor Analysis (CFA) was conducted to examine the factor structure of the mathematics affective characteristics model. DFA model results show that model data fit is reached (RMSEA=0.046, SRMR=0.051, CFI=0.973 and TLI=0.975). In the measurement invariance analysis, it was tested hierarchically between the stages with Multi-Group CFA (MG-CFA) analysis. The findings show that the mathematics affective characteristics model meets the configural, metric, scaler, and strict invariance stages, respectively. Therefore, the factor loadings, variances, error variances and covariances of the mathematics affective characteristics model were equivalent according to gender, and it was concluded that significant comparisons could be made between the groups. After examining measurement invariance, t-test analyses were conducted to examine the significant differences of the variables in the model according to gender. The results indicate that there is a significant difference in favor of boys in the MON scale, in favor of girls in the MKG and MDDD scales, while there is no significant difference in the MDV and MOS variables according to gender.

Keywords

• affective characteristics
• math
• motivation
• measurement invariance
• invariance

Matematik Duyuşsal Özellik Faktörlerinin Cinsiyete Göre Ölçme Değişmezliğinin İncelenmesi: TIMSS 2019 Türkiye Örneği

Öz

Uluslararası geniş ölçekli değerlendirmelerin temel hedeflerinden biri göre farklı ülkeler veya altgruplar arasında karşılaştırmalar yaparak eğitim politikaları veya eğitim sistemleri hakkında çıkarımlarda bulunmaktır. Farklı gruplar arasında karşılaştırma yapmanın temel kriterlerinden biri de ölçme değişmezliğinin sağlanmasıdır. Ölçme değişmezliği, ölçülen yapının gruplar arasında psikometrik olarak eşdeğer olduğunu göstermektedir. Ölçme değişmezliği kanıtı sunulmadan yapılan karşılaştırmalardaki farklılıklara dair iddialar güvenilmez olabilir. Bu çalışmanın amacı matematik duyuşsal özellikleri ile oluşturulan modelin cinsiyete göre ölçme değişmezliğinin sınanmasıdır. Bu amaçla TIMSS 2019 döngüsünde yer alan matematik öğrenmeyi sevme (MOS), matematik öğretiminin netliği (MON), matematik dersinde disiplinsiz davranış (MDDD), matematikte kendine güven (MKG) ve matematiğe değer verme (MDV) ölçekleri ile Matematik Duyuşsal Özellikleri Modeli oluşturulmuştur. Çalışmanın örneklemini TIMSS 2019 döngüsüne 8. Sınıf düzeyinde Türkiyeden katılan 3658 öğrenci oluşturmaktadır. Araştırmanın ilk bölümünde matematik duyuşsal özellikler modelinin faktör yapısını incelemek için Doğrulayıcı Faktör Analizi (DFA) yapılmıştır. DFA modeli sonuçları model veri uyumunun sağlandığını göstermektedir (RMSEA=0.046, SRMR=0.051, CFI=0.973 ve TLI=0.975). Ölçme değişmezliği analizinde Çok Gruplu DFA (ÇG-DFA) analizi ile aşamalar arasında hiyerarşik olarak test edilmiştir. Bulgular, matematik duyuşsal özellikler modelinin sırasıyla yapısal, metrik, ölçek ve katı değişmezlik aşamalarını karşıladığını göstermektedir. Dolayısıyla matematik duyuşsal özellikler modelinin cinsiyete göre faktör yükleri, varyansları, hata varyansları ve kovaryansları eşdeğer olup gruplar arasında anlamlı karşılaştırmalar yapılabileceği sonucuna ulaşılmıştır. Ölçme değişmezliğinin incelenmesinin ardından modelde yer alan değişkenlerin cinsiyete göre anlamlı farklılıklarını incelemek için t testi analizleri gerçekleştirilmiştir. Sonuçlar, MON ölçeğinde erkekler lehine, MKG ve MDDD ölçeklerinde kızlar lehine anlamlı farklılık olduğuna işaret ederken, MDV ve MOS değişkenlerinde cinsiyete göre anlamlı farklılık bulunmamaktadır.

Anahtar Kelimeler

• duyuşsal özellikler
• matematik
• motivasyon
• ölçme değişmezliği
• değişmezlik

References

1. Akben-Selcuk, E. (2017). Personality, motivation, and math achievement among Turkish students: Evidence from PISA data. Perceptual and Motor Skills, 124(2), 514–530. https://doi.org/10.1177/0031512516686505
2. Alatlı, B. (2020). Cross-cultural measurement invariance of the items in the science literacy test in the Programme for International Student Assessment (PISA-2015). International Journal of Education and Literacy Studies, 8(2), 16. https://doi.org/10.7575/aiac.ijels.v.8n.2p.16
3. Arseven, D. A. (1986). Çocukta Benlik Gelişimine Ailenin Etkisi ve Çocuğun Okuldaki Başarısı. Eğitim ve Bilim. 10 (60), 11-17.
4. Aybek E.C. (2022). Doğrulayıcı Faktör Analizi. Göçer Şahin S. ve Buluş, M. (Ed.), Adım Adım Uygulamalı İstatistik içinde (343-372). Pegem Yayınevi.
5. Bağdu Söyler, P., Aydin, B. ve Atilgan, H. (2021). PISA 2015 reading test item parameters across language groups: A measurement invariance study with binary variables. Egitimde ve Psikolojide Olcme ve Degerlendirme Dergisi, 112–128. https://doi.org/10.21031/epod.800697
6. Başusta, N. B. ve Gelbal, S. (2015). Gruplararası karşılaştırmalarda ölçme değişmezliğinin test edilmesi: PISA öğrenci anketi örneği. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi, 30(4), 80-90.
7. Bofah, E. A.-T. ve Hannula, M. S. (2015). TIMSS data in an African comparative perspective: Investigating the factors influencing achievement in mathematics and their psychometric properties. Large-Scale Assessments in Education, 3(1). https://doi.org/10.1186/s40536-015-0014-y
8. Brown, T.A. (2006). Confirmatory factor analysis for applied research. New York: Guilford Press.

9. Büyüköztürk, Ş. (2011). Sosyal Bilimler İçin Veri Analizi El Kitabı, 14. Baskı, Ankara: Pegem yayıncılık.
10. Büyüköztürk, Ş., Çakmak, E. K., Akgün, Ö. E., Karadeniz, Ş. ve Demirel, F. (2017). Bilimsel araştırma yöntemleri. Pegem, 1-360.
11. Cardoso, M. E. (2020). Policy evidence by design: International large-scale assessments and grade repetition. Comparative Education Review, 64(4), 598-618.
12. Cheung, G. W. ve Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural equation modeling, 9(2), 233-255.
13. Çakici Eser, D. (2021). Investigation of measurement invariance according to home resources: TIMSS 2015 mathematical affective characteristics questionnaire. International Journal of Assessment Tools in Education, 633–648. https://doi.org/10.21449/ijate.817168
14. Çiftçi, Ş. K. ve Yıldız, P. (2019). The Effect of Self-Confidence on Mathematics Achievement: The Meta-Analysis of Trends in International Mathematics and Science Study (TIMSS). International Journal of Instruction, 12(2), 683-694. https://doi.org/10.29333/iji.2019.12243a
15. Ding, Y., Yang Hansen, K. ve Klapp, A. (2022). Testing measurement invariance of mathematics self-concept and self-efficacy in PISA using MGCFA and the alignment method. European Journal of Psychology of Education. https://doi.org/10.1007/s10212-022-00623-y
16. Engel, L. C. ve Rutkowski, D. (2021). Costs of big data. In Digital Disruption In Teaching And Testing (pp. 124–135). Routledge.
17. Ersozlu, Z., Usak, M. ve Blake, D. (2022). Using Multi-Group Invariance analysis in exploring cross-cultural differences in mathematics anxiety: A comparison of Australia and Russia. Journal of Ethnic and Cultural Studies, 9(1), 1–18. https://doi.org/10.29333/ejecs/987
18. Ertürk, Z. ve Erdinç-Akan, O. (2018). TIMSS 2015 matematik başarısı ile ilgili bazı değişkenlerin cinsiyete göre ölçme değişmezliğinin incelenmesi. Journal of Theoretical Educational Science, 204-226.
19. F. Hair, J., Jr, Sarstedt, M., Hopkins, L. ve G. Kuppelwieser, V. (2014). Partial least squares structural equation modeling (PLS-SEM): An emerging tool in business research. European Business Review, 26(2), 106–121. https://doi.org/10.1108/ebr-10-2013-0128
20. Forero, C. G., Maydeu-Olivares, A. ve Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 625–641. https://doi.org/10.1080/10705510903203573
21. Fornell, C. ve Larcker, D. F. (1981). Structural equation models with unobservable variables and measurement error: Algebra and statistics. JMR, Journal of Marketing Research, 18(3), 382–388. https://doi.org/10.1177/002224378101800313
22. Guo, J., Parker, P. D., Marsh, H. W. ve Morin, A. J. S. (2015). Achievement, motivation, and educational choices: A longitudinal study of expectancy and value using a multiplicative perspective. Developmental Psychology, 51(8), 1163–1176. https://doi.org/10.1037/a0039440
23. Gustafsson, J.-E. (2018). International large scale assessments: Current status and ways forward. Scandinavian Journal of Educational Research, 62(3), 328–332. https://doi.org/10.1080/00313831.2018.1443573
24. Güngör, M. ve Atalay Kabasakal, K. (2020). Investigation of measurement invariance of science motivation and self-efficacy model: PISA 2015 turkey sample. International Journal of Assessment Tools in Education, 207–222. https://doi.org/10.21449/ijate.730481
25. Güvendir, M. A. (2016). Students' extrinsic and intrinsic motivation level and its relationship with their mathematics achievement. International Journal for Mathematics Teaching and Learning, 17(1).
26. He, J., Barrera-Pedemonte, F. ve Buchholz, J. (2019). Cross-cultural comparability of noncognitive constructs in TIMSS and PISA. Assessment in Education Principles Policy and Practice, 26(4), 369–385. https://doi.org/10.1080/0969594x.2018.1469467
27. Hooper, M., Mullis, I. V., Martin, M. O. ve Fishbein, B. (2020). TIMSS 2019 context questionnaire framework. Timss, 59-78.
28. Horn, J. L. ve McArdle, J. J. (1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18(3–4), 117–144. https://doi.org/10.1080/03610739208253916
29. Hu, L.-T. ve Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
30. Jöreskog, K. G. ve Sörbom, D. (1993). LISREL 8: Structural equation modeling with the SIMPLIS command language. Scientific software international.
31. Kam, C. C. S. (2019). Careless responding threatens factorial analytic results and construct validity of personality measure. Frontiers in Psychology, 10, 1258. https://doi.org/10.3389/fpsyg.2019.01258
32. Kam, C. C. S. ve Meyer, J. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18(3), 512–541. https://doi.org/10.1177/1094428115571894
33. Kıbrıslıoğlu, N. (2015). The investigation of measurement invariance PISA 2012 mathematics learning model according to culture and gender: Turkey - China (Shangai) – Indonesia [Master Thesis] Hacettepe University.
34. Kline, R. B. 2011. “Convergence of Structural Equation Modeling and Multilevel Modeling.” In The SAGE Handbook of Innovation in Social Research Methods, edited by M. Williams and W. P. Vogt, 562–589. SAGE Publications. doi:10.4135/9781446268261.
35. Koğar, H. ve Yilmaz Koğar, E. (2015). Comparison of different estimation methods for categorical and ordinal data in confirmatory factor analysis. Egitimde ve Psikolojide Olcme ve Degerlendirme Dergisi, 6(2). https://doi.org/10.21031/epod.94857
36. Ma, X. (1997). Reciprocal relationships between attitude toward mathematics and achievement in mathematics. The Journal of Educational Research, 90(4), 221–229. https://doi.org/10.1080/00220671.1997.10544576
37. Malone, T. W. ve Lepper, M. R. (2021). Making learning fun: A taxonomy of intrinsic motivations for learning. In Aptitude, learning, and instruction (pp. 223-254). Routledge.
38. Marsh, H. W. ve Craven, R. G. (2006). Reciprocal effects of self-concept and performance from a multidimensional perspective: Beyond seductive pleasure and unidimensional perspectives. Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 1(2), 133–163. https://doi.org/10.1111/j.1745-6916.2006.00010.x
39. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525–543.
40. Milfont, T. L. ve Fischer, R. (2010). Testing measurement invariance across groups: Applications in cross-cultural research. International Journal of psychological research, 3(1), 111-130.
41. Milli Eğitim Bakanlığı (2020). TIMSS 2019 Türkiye Ön Raporu. URL : https://odsgm.meb.gov.tr/meb_iys_dosyalar/2020_12/10175514_TIMSS_2019_Turkiye_On_Raporu_.pdf
42. Millsap, R. E. ve Olivera-Aguilar, M. (2012). Investigating measurement invariance using confirmatory factor analysis. In R. H. Hoyle, (Ed.) Handbook of structural equation modeling, (pp. 380-392), Guilford.
43. Mullis, I. V. S. ve Martin, M. O. (Eds.). (2017). TIMSS 2019 Assessment Frameworks. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: http://timssandpirls.bc.edu/timss2019/frameworks/
44. Mullis, I. V. S., Martin, M. O., Foy, P., Kelly, D. L. ve Fishbein, B. (2020). TIMSS 2019 International Results in Mathematics and Science. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: https://timssandpirls.bc.edu/timss2019/international-results/
45. Mullis, I. V. ve Martin, M. O. (2019). PIRLS 2021 Assessment Frameworks. International Association for the Evaluation of Educational Achievement. Herengracht 487, Amsterdam, 1017 BT, The Netherlands.
46. OECD (2013) PISA 2012 results: excellence through equity: giving every student the chance to succeed, vol II. PISA, OECD Publishing, Paris
47. Polat, M. (2019). TIMSS-2015 Matematik Ve Fen Duyuşsal Özellik Modellerinin Kültürlere, Cinsiyete ve Bölgelere Göre Ölçme Değişmezliğinin İncelenmesi. Ankara: Hacettepe Üniversitesi.
48. Putnick, D. L. ve Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review: DR, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004
49. Radovan, M. ve Makovec, D. (2015). Relations between students’ motivation, and perceptions of the learning environment. Center for Educational Policy Studies Journal, 5(2), 115–138. https://doi.org/10.26529/cepsj.145
50. Raykov, T. (2004). Behavioral scale reliability and measurement invariance evaluation using latent variable modeling. Behavior Therapy, 35(2), 299–331. https://doi.org/10.1016/s0005-7894(04)80041-8
51. Reynolds, K., Khorramdel, L. ve von Davier, M. (2022). Can students’ attitudes towards mathematics and science be compared across countries? Evidence from measurement invariance modeling in TIMSS 2019. Studies in Educational Evaluation, 74(101169), 101169. https://doi.org/10.1016/j.stueduc.2022.101169
52. Rosseel, Y. (2012). lavaan: AnRPackage for Structural Equation Modeling. Journal of Statistical Software, 48(2). https://doi.org/10.18637/jss.v048.i02
53. Rutkowski, L. ve Svetina, D. (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74(1), 31–57. https://doi.org/10.1177/0013164413498257
54. Ryan RM, Deci EL (2009) Promoting self-determined school engagement: motivation, learning, and well-being. In: Wentzel KR, Wigfield A (eds) Handbook on motivation at school. Routledge, New York, pp 171–196.
55. Schermelleh-Engel, K., Moosbrugger, H. ve Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of psychological research online, 8(2), 23-74.
56. Schleicher, A. (2019). PISA 2018: Insights and Interpretations. OECD Publishing.
57. Schmitt, N. ve Kuljanin, G. (2008). Measurement invariance: Review of practice and implications. Human Resource Management Review, 18(4), 210–222. https://doi.org/10.1016/j.hrmr.2008.03.003
58. Schreiber, J. B., Nora, A., Stage, F. K., Barlow, E. A. ve King, J. (2006). Reporting structural equation modeling and confirmatory factor analysis results: A review. The Journal of Educational Research, 99(6), 323–338. https://doi.org/10.3200/joer.99.6.323-338
59. Schumacker, R. E. ve Lomax, R. G. (2004). A beginner's guide to structural equation modeling. Psychology Press.
60. Şeyma, Uyar. ve Doğan, N. (2014). PISA 2009 Türkiye örnekleminde öğrenme stratejileri modelinin farklı gruplarda ölçme değişmezliğinin incelenmesi. Uluslararası Türk Eğitim Bilimleri Dergisi, 2014(3), 30-43.
61. Tabachnick, B. G. & Fidell, L. S. (2013). Using multivariate statistics (6th edition). Northridge: Pearson.
62. Taris, T. W., Bok, I. A. ve Meijer, Z. Y. (1998). Assessing stability and change of psychometric properties of multi-item concepts across different situations: A general approach. The Journal of Psychology, 132(3), 301–316. https://doi.org/10.1080/00223989809599169
63. Tavani C.M., Losh S.C. (2003) Motivation, self-confidence, and expectations as predictors of the academic performances among our high school students. Child Study J 33(3):141–151.
64. Teki̇n, Y. T. ve Çobanoğlu Aktan, D. (2021). Investigation of measurement invariance of PISA 2015 collaborative problem solving skills: Turkey, Norway and Singapore. International Journal of Assessment Tools in Education, 90–105. https://doi.org/10.21449/ijate.690576
65. Thien, L. M. ve Ong, M. Y. (2015). Malaysian and Singaporean students’ affective characteristics and mathematics performance: evidence from PISA 2012. Springer Plus, 4(1), 563. https://doi.org/10.1186/s40064-015-1358-z
66. Uyar, Ş. ve Kaya Uyanık, G. (2019). Fen Bilimlerine Yönelik Öğrenme Modelinin Ölçme Değişmezliğinin İncelenmesi: PISA 2015 Örneği. Kastamonu Eğitim Dergisi, 497–507. https://doi.org/10.24106/kefdergi.2570
67. Uyar, Y. D. D. Ş. (2021). Factor structure and measurement invariance of the TIMSS 2015 mathematics attitude questionnaire: Exploratory structural equation modelling approach. International Journal of Assessment Tools in Education, 8(4), 855–871. https://doi.org/10.21449/ijate.796862
68. Van De Schoot, R., Schmidt, P., De Beuckelaer, A., Lek, K. ve Zondervan-Zwijnenburg, M. (2015). Editorial: Measurement invariance. Frontiers in Psychology, 6, 1064. https://doi.org/10.3389/fpsyg.2015.01064
69. Van der Bergh, E. (2013). The influence of academic self–confidence on mathematics achievement (Doctoral dissertation, North-West University).
70. Vandenberg, R. J. ve Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational research methods, 3(1), 4-70.
71. Vandenberg, R. J. ve Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70. https://doi.org/10.1177/109442810031002
72. Woods, C. M. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28(3), 186-191.
73. Wu, A. D., Li, Z. ve Zumbo, B. D. (2007). Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analysis: A demonstration with TIMSS data. University of Massachusetts Amherst. https://doi.org/10.7275/MHQA-CD89
74. Yagan, S. A. (2021). The relationships between instructional clarity, classroom management and mathematics achievement: Mediator role of attitudes towards mathematics. University of South Florida M3 Center Publishing, 3(2021), 7.
75. Yandı, A., Köse, İ. A. ve Uysal, Ö. (2017). Farklı yöntemlerle ölçme değişmezliğinin incelenmesi: PISA 2012 örneği. Mersin Üniversitesi Eğitim Fakültesi Dergisi, 13(1), 243-253.
76. Yi̇ği̇ter, M. S. (2019). Öğretmenlerin teknoloji kullanımının mesleki motivasyonlarına etkisi: Çankaya ilçesi örneği. Yayımlanmamış Yüksek Lisans Tezi, Ankara Hacı Bayram Veli Üniversitesi, Lisansüstü Eğitim Enstitüsü.
77. Yin, L. ve Fishbein, B. (2019). Creating and interpreting the TIMSS 2019 context questionnaire scales. Methods and procedures: TIMSS, 16-1.
78. Zeng, D., Takada, N., Hara, Y., Sugiyama, S., Ito, Y., Nihei, Y. ve Asakura, K. (2022). Impact of intrinsic and extrinsic motivation on work engagement: A cross-sectional study of nurses working in long-term care facilities. International Journal of Environmental Research and Public Health, 19(3), 1284. https://doi.org/10.3390/ijerph19031284