Factor structure and measurement invariance of the TIMSS 2015 mathematics attitude questionnaire: Exploratory structural equation modelling approach

Abstract

In the current study, the appropriateness of the Mathematics Attitude Questionnaire administered to middle school 8th grade students in the TIMSS 2015 application to the exploratory structural equation and confirmatory factor analysis models was examined. The study was conducted on 6079 students making up the sample of Turkey. In the TIMSS 2015 application, the attitude items are presented under four headings called students’ interest in mathematics, students’ views on engaging teaching in mathematics lessons, students’ self-confidence in mathematics, and students’ value mathematics. As a result of the investigation of the factor structure of these items, the attitude questionnaire with its 5 factors and 35 items was accepted to be suitable for the Exploratory Structural Equation Model (ESEM). Moreover, invariance of the TIMSS 8th grade mathematics attitude questionnaire depending on gender was investigated at six stages as configural, weak (metric), strong (scalar), strict, variance-covariance, and latent mean invariance through ESEM. It was concluded that the questionnaire satisfied all the invariance conditions.

Keywords

• Exploratory structural equation model
• Measurement invariance
• TIMSS
• Multi-group confirmatory factor analysis

References

1. Asparouhov, T., & Muthén, B. (2006). Robust chi square difference testing with mean and variance adjusted test statistics. Mplus Web Notes, 10.
2. Asparouhov, T., & Muthen, B. (2009). Exploratory structural equation modeling. Structural Equation Modeling, 16, 397–438. https://doi.org/10.1080/10705510903008204
3. Allison, P. D. (2003). Missing data techniques for structural equation modeling. Journal of Abnormal Psychology, 112(4), 545. https://doi.org/10.1037/0021-843X.112.4.545
4. Asparouhov, T; Muthén, B. (2010). Computing the strictly positive Satorra-Bentler chi-square test in Mplus. Mplus Web Notes, 12, 1-12.
5. Başusta, N. B., & Gelbal, S. (2015). Gruplararası karşılaştırmalarda ölçme değişmezliğinin test edilmesi: PISA öğrenci anketi örneği [Examination of measurement invariance at groups' comparisons: A study on PISA student questionnaire]. Hacettepe University Journal of Education, 30(4), 80 90. http://www.efdergi.hacettepe.edu.tr/yonetim/icerik/makaleler/1773-published.pdf
6. Bofah, E. A. T., & Hannula, M. S. (2015). TIMSS data in an African comparative perspective: Investigating the factors influencing achievement in mathematics and their psychometric properties. Large Scale Assessments in Education, 3(1), 1 36. http://dx.doi.org/10.1186/s40536-015-0014-y
7. Booth, T., & Hughes, D. J. (2014). Exploratory structural equation modeling of personality data. Assessment, 21(3), 260-271. https://doi.org/10.1177/1073191114528029
8. Bornstein, M. H. (1995). Form and function: Implications for studies of culture and human development. Culture & Psychology, 1(1), 123 137. https://doi.org/10.1177/1354067X9511009

9. Brown, T. (2006). Confirmatory factor analysis for applied research. The Guilford Press.
10. Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen and J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Sage.
11. Bryant, F. B., & Satorra, A. (2012). Principles and practice of scaled difference chi-square testing. Structural Equation Modeling: A Multidisciplinary Journal, 19(3), 372-398. https://doi.org/10.1080/10705511.2012.687671
12. Büyüköztürk, Ş. (2002). Faktör analizi: Temel kavramlar ve ölçek geliştirmede kullanımı [Factor analysis: Basic concepts and using to development scale]. Educational Administration in Theory and Practice, 8(4), 470 483. https://dergipark.org.tr/tr/pub/kuey/issue/10365/126871
13. Caro, D. H., Sandoval-Hernández, A., & Lüdtke, O. (2014). Cultural, social, and economic capital constructs in international assessments: An evaluation using exploratory structural equation modeling. School Effectiveness and School Improvement, 25(3), 433-450. https://doi.org/10.1080/09243453.2013.812568
14. Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464 504. https://doi.org/10.1080/10705510701301834
15. Chung, H., Kim, J., Park, R., Bamer, A. M., Bocell, F. D., & Amtmann, D. (2016). Testing the measurement invariance of the University of Washington Self-Efficacy Scale short form across four diagnostic subgroups. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 25(10), 2559–2564. https://doi.org/10.1007/s11136-016-1300-z
16. Cudeck, R., & MacCallum, R. C. (Eds.). (2007). Factor analysis at 100: Historical developments and future directions. Lawrence Erlbaum
17. Çokluk, Ö., Şekercioğlu, G. & Büyüköztürk, Ş. (2010). Sosyal bilimler için çok değişkenli istatistik: SPSS ve LISREL uygulamaları. Pegem Akademi.
18. Ertürk, Z., & Erdinç-Akan, O. (2018). TIMSS 2015 matematik başarısı ile ilgili bazı değişkenlerin cinsiyete göre ölçme değişmezliğinin incelenmesi [The investigation of measurement invariance of the variables related to TIMSS 2015 mathematics achievement in terms of gender]. Journal of Theoretical Educational Science, 204-226. https://dergipark.org.tr/tr/pub/akukeg/issue/40520/412604
19. Gomes, C., Almeida, L., & Nunez, J. (2017). Rationale and Applicability of Exploratory Structural Equation Modeling (ESEM) in psychoeducational contexts. Psicothema, 29(3), 396-401.
20. Graham, J. W. (2012). Missing data: Analysis and design. Springer Science & Business Media.
21. Guay, F., Morin, A., Litalien, D., Valois, P., & Vallerand, R. (2015). Application of Exploratory Structural Equation Modeling to Evaluate the Academic Motivation Scale. The Journal of Experimental Education, 83(1), 51-82. https://doi.org/10.1080/00220973.2013.876231
22. Guo, J. M. (2019). A Systematic evaluation and comparison between exploratory structural equation modeling and bayesian structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 26, 529-556. https://doi.org/10.1080/10705511.2018.1554999
23. Guo, J., Parker, H., Dicke, P., Lüdtke, T., & Diallo, T. (2019). A systematic evaluation and comparison between exploratory structural equation modeling and bayesian structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 26 (4), 529-556. https://doi.org/10.1080/10705511.2018.1554999
24. Güngör, M & Atalay Kabasakal, K. (2020). Investigation of measurement invariance of science motivation and self-efficacy model: PISA 2015 Turkey sample. International Journal of Assessment Tools in Education, 7(2), 207-222. https://doi.org/10.21449/ijate.730481
25. Horn, J. L., & McArdle, J. J. (1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18(3), 117-144. https://doi.org/10.1080/03610739208253916
26. Hu, L., & Bentler, P. (1995). Evaluating model fit. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 76−99). Sage.
27. Jennrich, R. I. and Sampson, P. F. (1966). Rotation to simple loadings. Psychometrika, 31(3), 313–323. https://link.springer.com/article/10.1007/BF02289465
28. Jöreskog, K.G. (1969). A general approach to confirmatory maximum likelihood factor analysis. Psychometrika, 34, 183–202.
29. Joshanloo, M., & Lamers, S. M. (2016). Reinvestigation of the factor structure of the MHC-SF in the Netherlands: Contributions of exploratory structural equation modeling. Personality and Individual Differences, 97, 8 12. https://doi.org/10.1016/j.paid.2016.02.089
30. Jung, J. Y. (2019): A Comparison of CFA and ESEM approaches using TIMSS science attitudes items: evidence from factor structure and measurement invariance. [Master’s Thesis, Purdue University]. Purdue University Graduate School, Department of Educational Studies, https://doi.org/10.25394/PGS.7995890.v1
31. Kıbrıslıoğlu, N. (2015). The investigation of measurement invariance PISA 2012 mathematics learning model according to culture and gender: Turkey-China (Shangai)-Indonesia [Master's Thesis]. Hacettepe University.
32. Kline, R. B. (2005). Methodology in the social sciences. Principles and practice of structural equation modeling (2nd ed.). Guilford Press.
33. Kristjansson, S. D., Pergadia, M. L., Agrawal, A., Lessov- Schlaggar, C. N., McCarthy, D. M., Piasecki, T. M. & Heath, A. C. (2011). Smoking outcome expectancies in young adult female smokers: Individual differences and associations with nicotine dependence in a genetically informative sample. Drug and Alcohol Dependence, 116, 37 44. https://doi.org/10.1016/j.drugalcdep.2010.11.017
34. Krueger, R. F., Markon, K. E., Patrick, C. J., Benning, S. D., & Kramer, M. D. (2007). Linking antisocial behavior, substance use, and personality: an integrative quantitative model of the adult externalizing spectrum. Journal of Abnormal Psychology, 116(4), 645. https://doi.org/10.1037/0021-843X.116.4.645
35. Little, T. D. (2013). Longitudinal structural equation modeling. Guilford press.
36. Little, R. J., & Rubin, D. B. (1987). Statistical analysis with missing data. John Wiley & Sons.
37. Marsh, H. W., Abduljabbar, A. S., Abu-Hilal, M. M., Morin, A. J. S., Abdelfattah, F., Leung, K. C., Xu, M. K., Nagengast, B., & Parker, P. (2013). Factorial, convergent, and discriminant validity of timss math and science motivation measures: A comparison of Arab and Anglo-Saxon countries. Journal of Educational Psychology, 105(1), 108–128. https://doi.org/10.1037/a0029907
38. Marsh, H. W., Hau, K.-T., & Grayson, D. (2005). Goodness of Fit in Structural Equation Models. In A. Maydeu-Olivares & J. J. McArdle (Eds.), Multivariate applications book series. Contemporary psychometrics: A festschrift for Roderick P. McDonald (p. 275–340). Lawrence Erlbaum Associates Publishers.
39. Marsh, H. W., Liem, G. A. D., Martin, A. J., Morin, A. J., & Nagengast, B. (2011). Methodological measurement fruitfulness of exploratory structural equation modeling (ESEM): New approaches to key substantive issues in motivation and engagement. Journal of Psychoeducational Assessment, 29(4), 322 346. https://doi.org/10.1177/0734282911406657
40. Marsh, H. W., Lüdtke, O., Muthén, B., Asparouhov, T., Morin, A. J., Trautwein, U., & Nagengast, B. (2010). A new look at the big five factor structure through exploratory structural equation modeling. Psychological Aassessment, 22(3), 471. https://doi.org/10.1037/a0019227
41. Marsh, H. W., Morin, A. J., Parker, P. D., & Kaur, G. (2014). Exploratory structural equation modeling: An integration of the best features of exploratory and confirmatory factor analysis. Annual Review of Clinical Psychology, 10, 85 110. https://doi.org/10.1146/annurev-clinpsy-032813-153700
42. Marsh, H. W., Muthén, B., Asparouhov, T., Lüdtke, O., Robitzsch, A., Morin, A. J., & Trautwein, U. (2009). Exploratory structural equation modeling, integrating CFA and EFA: Application to students' evaluations of university teaching. Structural Equation Modeling: A Multidisciplinary Journal, 16(3), 439 476. https://doi.org/10.1080/10705510903008220
43. Mattsson, M. (2012). Investigating the factorial invariance of the 28-item DBQ across genders and age groups: an exploratory structural equation modeling study. Accident Analysis & Prevention, 48, 379-396. https://doi.org/10.1016/j.aap.2012.02.009
44. Ministry of National Education (2016). TIMSS 2015 ulusal matematik ve fen ön raporu: 4. ve 8. sınıflar [TIMSS 2015 national mathematics and sciences preliminary report 4th and 8th grades]. https://odsgm.meb.gov.tr/meb_iys_dosyalar/2017_06/23161945_timss_2015_on_raporu.pdf
45. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525-543. https://link.springer.com/article/10.1007/BF02294825
46. Morin, A. J. S., & Maïano, C. (2011). Cross-validation of the short form of the physical self-inventory (PSI-S) using exploratory structural equation modeling (ESEM). Psychology of Sport and Exercise. 12, 540–554. https://doi.org/10.1016/j.psychsport.2011.04.003
47. Nunnally, J.C. (1978). Psychometric theory (2nd ed.). McGraw-Hill
48. Özdamar, K. (2004). Paket Programlar ile istatistiksel veri analizi (Çok değişkenli analizler). Kaan Kitabevi.
49. Polat, M. (2019). The investigation of measurement invariance of TIMSS-2015 mathematics and science affective characteristics models according to culture, gender and statistical region [Master's Thesis], Hacettepe University.
50. Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71-90. https://doi.org/10.1016/j.dr.2016.06.004
51. Rencher, A. (2002). Methods of multivariate analysis. John Wiley & Sons, Inc.
52. Raykov, T., & Marcoulides, G. (2008). Introduction to applied multivariate analysis. Routledge Taylor & Francis Group.
53. Rosellini, A. J., & Brown, T. A. (2011). The NEO Five-Factor Inventory: Latent structure and relationships with dimensions of anxiety and depressive disorders in a large clinical sample. Assessment, 18(1), 27-38. http://dx.doi.org/10.1177/1073191110382848
54. Sass, D. A., Schmitt, T. A., & Marsh, H. W. (2014). Evaluating model fit with ordered categorical data with a measurement invariance framework: A comparison of estimators. Structural Equation Modeling, 21, 167 180. https://doi.org/10.1080/10705511.2014.882658
55. Satorra, A., & Bentler, P. M. (2010). Ensuring positiveness of the scaled difference chi-square test statistic. Psychometrika, 75(2), 243–248. http://dx.doi.org/10.1007/s11336-009-9135-y
56. Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of psychological research online, 8(2), 23-74. https://psycnet.apa.org/record/2003-08119-003
57. Schmitt, T. A. (2011). Current methodological considerations in exploratory and confirmatory factor analysis. Journal of Psychoeducational Assessment, 29(4), 304 321. https://doi.org/10.1177/0734282911406653
58. Simpkins, S. D., Davis-Kean, P. E., & Eccles, J. S. (2005). Parents' socializing behavior and children's participation in math, science, and computer out-of-school activities. Applied Developmental Science, 9(1), 14-30. https://doi.org/10.1207/s1532480xads0901_3
59. Stark, S., Chernshenko, O. S., & Drasgow, F. (2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91(6), 1292–1306.
60. Stapleton, C. (1997). Basic concepts and procedures of confirmatory factor analysis. Paper presented at the annual meeting of the Southwest Educational Research Association January 23-25. Austin.
61. Steenkamp, J. B. E., & Baumgartner, H. (1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25(1), 78-90.
62. Stromeyer, W. R., Miller, J. W., Sriramachandramurthy, R., & DeMartino, R. (2015). The prowess and pitfalls of Bayesian structural equation modeling: Important considerations for management research. Journal of Management, 41(2), 491 520. https://doi.org/10.1177/0149206314551962
63. Tabachnick, B. G., Fidell, L. S., & Ullman, J. B. (2007). Using multivariate statistics (Vol. 5, pp. 481-498). Pearson.
64. Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. American Psychological Association.
65. Tomás, I., Marsh, H. W., González-Romá, V., Valls, V., & Nagengast, B. (2014). Testing measurement invariance across Spanish and English versions of the Physical Self-Description Questionnaire: An application of exploratory structural equation modeling. Journal of Sport and Exercise Psychology, 36(2), 179 188. https://doi.org/10.1123/jsep.2013-0070
66. Uyar, Ş, Kaya Uyanık, G. (2019). Fen bilimlerine yönelik öğrenme modelinin ölçme değişmezliğinin incelenmesi: PISA 2015 örneği [Investigating measurement invariance of learning model towards science: PISA 2015 sample] Kastamonu Education Journal, 27(2), 497-507. https://doi.org/10.24106/kefdergi.2570
67. Watt, H. M. (2004). Development of adolescents' self‐perceptions, values, and task perceptions according to gender and domain in 7th‐through 11th‐grade Australian students. Child Development, 75(5), 1556-1574.
68. Whitaker, B. G., & McKinney, J. L. (2007). Assessing the measurement invariance of latent job satisfaction ratings across survey administration modes for respondent subgroups: A MIMIC modelling approach. Behavior Research Methods, 39(3), 502 509. https://doi.org/10.3758/BF03193019