Investigation of Measurement Invariance of Science Motivation and Self-Efficacy Model: PISA 2015 Turkey Sample

Abstract

Measurement invariance analyses are carried out in order to find evidence for the structural validity of the measurement tools used in the field of educational sciences and psychology. The purpose of this research is to examine the measurement invariance of Science Motivation and Self-Efficacy Model constructed by Instrumental Motivation to Learn Science and Science Self-Efficacy subscales found in the PISA 2015 Student Questionnaire across different groups in the Turkish sample survey. The analysis was carried out with the data obtained from 4583 students that met the analysis assumptions. The measurement invariance of the model in terms of gender and statistical regional groups was examined by the structural equation modeling (SEM) technique. Firstly, the data was examined to determine whether the assumptions for the analyses were met. Then, measurement models were verified by performing confirmatory factor analysis (CFA). The measurement invariance across genders and statistical regions was tested by multi-group confirmatory factor analysis (MGCFA). Unweighted Least Squares (ULS) method was used as the estimation method in CFA and MGCFA stages. In order to make final decisions about the stage of measurement invariance models hold, Comparative Fit Index (CFI) was used. The results of the study show that the research model ensures all stages of invariance across gender groups and regions. Science Motivation and Self-Efficacy Model illustrates that it is valid to make comparisons between scores of male and female students or students from different regions of Turkey. According to the findings, the research model could provide complete measurement invariance.

Keywords

• Structural equation modeling
• Measurement invariance
• Instrumental motivation
• Science self-efficacy

References

1. Ağaç, G. & Masal, E. (2015). An investigation of the relation between 8th grade students’ beliefs, abstract thought and achievement; The case of mathematics. International Online Journal of Educational Sciences, 7(1), 134-144.
2. Akyıldız, M. (2009). PIRLS 2001 testinin yapı geçerliliğinin ülkelerarası karşılaştırılması [The comparison of construct validities of the PIRLS 2001 test between countries]. Van Yuzuncu Yil University Journal of Education, 6(1), 18-47.
3. Alivernini, F. (2011). Measurement invariance of a reading literacy scale in the Italian context: A psychometric analysis. Procedia Social and Behavioral Sciences, 15, 436-441.
4. Anderson, C. J. & Gerbing, D. W. (1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 3, 411-423.
5. Asil, M. & Gelbal, S. (2012). PISA öğrenci anketinin kültürler arası eşdeğerliği [Cross-cultural equivalence of the PISA student questionnaire]. Education and Science, 37(166), 236-249.
6. Ayvallı, M. & Biçak, B. (2018). An investigation into the measurement invariance of PISA 2012 mathematical literacy test. European Journal of Education Studies, 4 (11), 39-58.
7. Bakan Kalaycıoğlu, D. (2015). The influence of socioeconomic status, self-efficacy, and anxiety on mathematics achievement in England, Greece, Hong Kong, the Netherlands, Turkey, and the USA. Educational Sciences: Theory & Practice, 15(5), 1-11.
8. Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: W. H. Freeman.

9. Başusta, B. N. (2010). Ölçme eşdeğerliği [Measurement equivalence]. Journal of Measurement and Evaluation in Education and Psychology, 1(2), 58-64.
10. Başusta, B. N. & Gelbal, S. (2015). Gruplararası karşılaştırmalarda ölçme değişmezliğinin test edilmesi: PISA anketi örneği [Examination of measurement invariance at groups’ comparisons: A study on PISA student questionnaire]. Hacettepe University Journal of Education, 30(4), 80-90.
11. Batyra, A. (2017). Gender gaps in student achievement in Turkey: Evidence from Trends in International Mathematics and Science Study (TIMSS) 2015. Education Reform Initiative & Aydın Doğan Foundation.
12. Bıkmaz, H. F. (2002). Fen öğretiminde öz-yeterlik inancı ölçeği [Self-efficacy belief instrument in science teaching]. Educational Sciences & Practice, 1(2), 197-210.
13. Bircan, H. & Sungur, S. (2016). The role of motivation and cognitive engagement in science achievement. Science Education International, 27(4), 509-529.
14. Boyd, L. B., Dooley, E. K. & Felton, S. (2006). Measuring learning in the affective domain using reflective writing about a virtual international agricultural experience. Journal of Agricultural Education, 47(3), 24-32.
15. Britner, S. L. & Pajares, F. (2001). Self-efficacy beliefs, motivation, race, and gender in middle school science. Journal of Women and Minorities in Science and Engineering, 7(4), 271-285.
16. Browne, M. W. & Cudeck, R. (1993). Alternative ways of assessing model fit, testing structural equation models, K. A. Bollen & J. S. Long (Eds.), Newbury Park, CA: Sage.
17. Byrne, B. M., Shavelson, R. J. & Muthen, B. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105(3), 456-466.
18. Byrne, B. M. & Watkins, D. (2003). The issue of measurement invariance revisited. Journal of Cross-cultural Psychology, 34(2), 155-175.
19. Cheung, G. W. & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255.
20. Çokluk, Ö., Şekercioğlu, G. & Büyüköztürk, Ş. (2016). Sosyal bilimler için çok değişkenli istatistik: SPSS ve LISREL uygulamaları. Ankara: Pegem.
21. Dilts, R. (1998). Motivation. Retrieved August 17, 2019 from http://nlpu.com/Articles/artic17.htm
22. Ercikan, K. & Koh, K. (2005). Examining the construct comparability of the English and French versions of TIMSS. International Journal of Testing, 5, 23-35.
23. Ertürk, Z. & Erdinç Akan, O. (2018). TIMSS 2015 matematik başarısı ile ilgili bazı değişkenlerin cinsiyete göre ölçme değişmezliğinin incelenmesi [The investigation of measurement invariance of the variables related to TIMSS 2015 mathematics achievement in terms of gender]. Journal of Theoritical Educational Science, UBEK-2018, 204-226.
24. Forero, G. C., Maydeu-Olivares, A. & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A monte carlo study comparing DWLS and ULS estimation. Structural Equation Modeling, 16, 625-641.
25. Gierl, M. J. (2000). Construct equivalence on translated achievement tests. Canadian Journal of Education, 25(4), 280-296.
26. Gregorich, S. E. (2006). Do self-report instruments allow meaningful comparisons across diverse population groups?: testing measurement invariance using the confirmatory factor analysis framework. Medical Care, 44(11), 78-94.
27. Gujarati, D. N. (1995). Basic econometrics (3rd Ed.). New York, NY: Mc-Graw Hill.
28. Gülleroğlu, D. H. (2017). PISA 2012 matematik uygulamasına katılan Türk öğrencilerin duyuşsal özelliklerinin cinsiyete göre ölçme değişmezliğinin incelenmesi [An investigation of measurement invariance by gender for the Turkish students’ affective characteristics who took the PISA 2012 math test]. Gazi University Journal of Gazi Educational Faculty, 37(1), 151-175.
29. Hambleton, R. K. (1994). Guidelines for adapting educational and psychological tests: A progress report. European Journal of Psychological Assessment, 10, 229–244.
30. Hirschfeld, G. & Brachel, v. R. (2014). Multiple-group confirmatory factor analysis in R – A tutorial in measurement invariance with continuous and ordinal indicators. Practical Assessment, Research & Evaluation, 19(7), 1-12.
31. Hu, L. & Bentler, M. P. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55.
32. İlhan, K. (2015). Eğitimde pozitif psikoloji uygulamaları. B. Ergüner Tekinalp & Ş. Işık (Ed.). Ankara: Pegem Akademi Yayıncılık.
33. İmrol, F. (2017). Investigation of measurement invariance of motivation and self-belief constructs towards mathematics in PISA 2012 Turkey sample (Master thesis). Retrieved August 14, 2019 from https://tez.yok.gov.tr/UlusalTezMerkezi/
34. Jöreskog, K. G. & Sörbom, D. (1999). LISREL 8 user’s reference guide. Chicago: Science Software International.
35. Kaplan, D. (2000). Structural equation modeling: Foundations and extensions. Newbury Park, CA: Sage.
36. Karakoç Alatlı, B., Ayan, C., Polat Demir, B. & Uzun, G. (2016). Examination of the TIMSS 2011 fourth grade mathematics test in terms of cross-cultural measurement invariance. Euroasian Journal of Educational Research, 66, 389-406.
37. Kıbrıslıoğlu, N. (2015). The investigation of measurement invariance PISA 2012 mathematics learning model according to culture and gender: Turkey – China (Shangai) - Indonesia (Master thesis). Retrieved August 14, 2019 from https://tez.yok.gov.tr/UlusalTezMerkezi/
38. Kıbrıslıoğlu Uysal, N. & Akın Arıkan, Ç. (2018). Measurement invariance of science self-efficacy scale in PISA. International Journal of Assessment Tools in Education, 5 (2), 325-338.
39. Kline, R. B. (2005). Principles and practices of structural equation modeling (2nd Ed.). New York: Guilford Press.
40. Kline, R. B. (2011). Principles and practices of structural equation modeling (3rd Ed.). New York: Guilford Press.
41. Koğar, H. & Yılmaz Koğar, E. (2015). Comparison of different estimation methods for categorical and ordinal data in confirmatory factor analysis. Journal of Measurement and Evaluation in Education and Psychology, 6(2), 351-364.
42. Koh, H. K. & Zumbo, D. B. (2008). Multi-group confirmatory factor analysis for testing measurement invariance in mixed item format data. Journal of Modern Applied Statistical Methods, 7(2), 471-477.
43. Larson, L. M., Stephen, A., Bonitz, V. S. & Wu, T.-F. (2014). Predicting science achievement in India: role of gender, selfefficacy, interests, and effort. Journal of Career Assessment, 22(1), 89-101.
44. Mellenberg, G. J. (1989). Item bias and item response theory. International Journal of Educational Research: Applications of Item Response Theory, 13(2), 123-144.
45. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525-543.
46. Meredith, W. & Millsap, E. R. (1992). On the misuse of manifest variables in the detection of measurement bias. Psychometrika, 57(2), 289-311.
47. Ministry of National Education [MNE]. (2016). PISA 2015 national report. Ankara.
48. Nagengast, B. & Marsh, H. (2014). Motivation and engagement in science around the globe: Testing measurement invariance with multigroup structural equation models across 57 countries using PISA 2006. In L. Rutkowski, M. von Davier & D. Rutkowski (Eds.), Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis (pp 317-345). UK: Taylor & Francis.
49. OECD [Organisation for Economic Co-operation and Development]. (2016). PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic and Financial Literacy. PISA, OECD, Paris: OECD Publishing.
50. OECD [Organisation for Economic Co-operation and Development]. (2017). PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic and Financial Literacy and Collaborative Problem Solving, revised edition. PISA, Paris: OECD Publishing.
51. Oliden, E. P. & Lizaso, M. J. (2013). Invariance levels across language versions of the PISA 2009 reading comprehension tests in Spain. Psicothema, 25(3), 390-395.
52. Ölçüoğlu, R. & Çetin, S. (2016). TIMSS 2011 sekizinci sınıf öğrencilerinin matematik başarısını etkileyen değişkenlerin bölgelere göre incelenmesi [The investigation of the variables that affecting eight grade students’ TIMSS 2011 math achievement according to regions]. Journal of Measurement and Evaluation in Education and Psychology, 7(1), 202-220.
53. Polat, E. & Madra, A. (2018). PISA 2015 ve TIMSS 2015 ışığında Türkiye’de cinsiyete dayalı başarı farkı. Education Reform Initiative & Aydın Doğan Foundation, 1-18.
54. Raykov, T. & Marcoulides, A. G. (2006). A first course in structural equation modeling. Lawrence Erlbaum Associates, Inc.
55. Reise, P. S., Widaman, F. K. & Pugh, H. R. (1993). Confirmatory factor analysis and item response theory: two approaches for exploring measurement invariance. Psychological Bulletin, 114(3), 552-566.
56. R Development Core Team (2017). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved August 14, 2019 from http://www.R-project.org/
57. Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36.
58. Scherer, R., Nilsen, T. & Jansen, M. (2016). Evaluating individual students’ perceptions of instructional quality: an investigation of their factor structure, measurement invariance, and relations to educational outcomes. Front. Psychol., 7(110), 1-16.
59. Schermelleh-Engel, K., Moosbrugger, H. & Müller, H. (2003). Evaluating the fit of structural equation models: Test of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23-74.
60. Schumacker, E. R. & Lomax, G. R. (1996). A beginner’s guide to structural equation modeling. Manwah, NJ: Lawrence Erlbaum Associates.
61. Schunk, D. H., Meece, J. L. & Pintrich, P. R. (2014). Motivation in education: Theory, research and applications. New Jersey: Pearson Education, Inc.
62. Schunk, D. H. & Pajares, F. (2009). Self-efficacy theory. In K. R. Wentzel & A. Wigfield (Eds.), Handbook of Motivation at School (pp. 35-53). New York, NY: Routledge.
63. Sireci, S. G., Patsula, L. N. & Hambleton, R. K. (2005). Statistical Methods for Identifying Flaws in the Test Adaptation Process. In R. K. Hambleton, P. F. Merenda & C. D. Spielberger (Eds.), Adapting Educational and Psychological Tests for Cross-Cultural Assessment. New Jersey, London: Lawrence Erlbaum Associates, Publishers.
64. Somer, O., Korkmaz, M., Dural, S. & Can, S. (2009). Ölçme eşdeğerliğinin yapısal eşitlik modellemesi ve madde cevap kuramı kapsamında incelenmesi [Detection of measurement equivalence by structural equation modeling and item response theory]. Turkish Journal of Psychology, 24(64), 61–75.
65. Steenkamp, J.-B. & Baumgartner, H. (1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78-90.
66. Tabachnick, B. G. & Fidell, L. S. (2013). Using multivariate statistics. Boston: Pearson.
67. Taris, W. T., Bok, A. I. & Meijer, Z. Y. (1998). Assessing stability and change of psychometric properties of multi-item concepts across different situations: a general approach. The Journal of Psychology, 132(3), 301-316.
68. Uyar, Ş. & Doğan, N. (2014). Türkiye örnekleminde öğrenme stratejileri modelinin farklı gruplarda ölçme değişmezliğinin incelenmesi [An investigation of measurement invariance of learning strategies model across different groups in PISA Turkey sample]. International Journal of Turkish Education Sciences, 3, 30-43.
69. Uyar, Ş. & Kaya Uyanık, G. (2019). Fen bilimlerine yönelik öğrenme modelinin ölçme değişmezliğinin incelenmesi: PISA 2015 örneği [Investigation measurement invariance of learning model towards science: PISA 2015 sample]. Kastamonu Education Journal, 27(2), 297-507.
70. Uzun, N. B., Gelbal, S. & Öğretmen, T. (2010). TIMMS-R fen başarısı ve duyuşsal özellikler arasındaki ilişkinin modellenmesi ve modelin cinsiyetler bakımından karşılaştırılması [Modeling the relationship between TIMSS-R science achievement and affective characteristics and comparing the model according to gender]. Kastamonu Education Journal, 18(2), 531-544.
71. Yandı, A., Köse, A. İ. & Uysal, Ö. (2017). Examining measurement invariance with different methods: Example of PISA 2012. Mersin University Journal of the Faculty of Education, 13(1), 243-253.
72. Vandenberg, J. R. & Lance, E. C. (2000). A review and synthesis of the measurement invariance literature: suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4-70.
73. Wigfield, A. & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation. Contemporary Educational Psychology, 25(1), 68-81.
74. Wu, A. D., Li, Z. & Zumbo, B. D. (2007). Decoding the meaning of factorial invariance and updating the practice of multigroup confirmatory factor analysis: a demonstration with TIMSS data. Practical Assessment, Research & Evaluation, 12(3), 1-26.
75. Zedlin, A. L., Britner, S. L. & Pajares, F. (2007). A comparative study of the self-efficacy beliefs of successful men and women in mathematics, science, and technology careers. Journal of Research in Science Teaching, 45(9), 1036–1058.