Investigation of measurement invariance of PISA 2015 collaborative problem solving skills Turkey, Norway, Singapore

Öz

Bu araştırmanın amacı, PISA 2015 Xandar alt testi ile ölçülen iş birlikli problem çözme becerilerinin Singapur, Norveç ve Türkiye için ölçme değişmezliğini incelemektir. Araştırma, Türkiye (1032), Norveç (923) ve Singapur'dan (1035) PISA 2015 iş birlikli problem becerileri çalışmasında 2990 katılımcıdan elde edilen veriler ile gerçekleştirilmiştir. Araştırmanın ilk adımında, Xandar alt testinin faktör yapısını elde etmek için açımlayıcı faktör analizi yapılmıştır. İkinci adımda ise elde edilen model doğrulayıcı faktör analizi ile incelenmiştir. Bunun için verilerin modele uyumu doğrulayıcı faktör analizinde2 / df (3.127), RMSEA (0.027), CFI (0.987) ve TLI (0.979) değerleri ile kontrol edilmiştir. İş birlikli problem çözme becerilerinin ölçme değişmezliğinin analizi için çok gruplu doğrulayıcı faktör analizi kullanılmıştır. Bulgular, işbirlikçi problem çözme modelinin Singapur, Norveç ve Türkiye için sadece yapısal değişmezliği sağladığını, ancak metrik, ölçek ve katı değişmezlik aşamalarını sağlamadığını göstermektedir. Elde edilen sonuçlar, bu ülkeler arasında anlamlı karşılaştırmalar yapılamayacağını, çünkü faktör yükleri, varyanslar, hata farkları ve kovaryansların ülkeler arasında farklılaştığını göstermektedir.

Anahtar Kelimeler

• PISA Collaborative problem solving skills, Measurement invariance
• Multi-group confirmatory factor analysis
• Cultural comparisons

Investigation of measurement invariance of PISA 2015 collaborative problem solving skills: Turkey, Norway and Singapore

Abstract

The purpose of this research is to examine measurement invariance of collaborative problem solving skills measured by PISA 2015 Xandar subtest for Singapore, Norway, and Turkey. The research was conducted with 2990 participants’ data obtained from Turkey (1032), Norway (923), and Singapore (1035) on PISA 2015 collaborative problem solving study. In the first part of the study, exploratory factor analysis was performed to obtain the factor structure of the Xandar subtest. Then, the model data fit was checked by confirmatory factor analysis via X²/df (3.127), RMSEA (0.027), CFI (0.987) and TLI (0.979) values. Multi-group confirmatory factor analysis was used in invariance analyses. The findings show that the collaborative problem solving model met only configural invariance across the countries and has not met the metric, scale, and strict invariance stages. The results show that meaningful comparisons cannot be made between the countries, because the factor loadings, variances, error variances, and covariances differ among countries.

Keywords

• PISA Collaborative problem solving skills
• Measurement invariance
• Multi-group confirmatory factor analysis
• Cultural comparisons

References

1. Bahadır, E. (2012). According programme for ınternational student assessment (PISA 2009), investigation of variables that affect Turkish students' reading skills by regions. Master Thesis, Hacettepe University, Ankara.
2. Başusta, N.B., & Gelbal, S. (2015). Examination of Measurement Invariance at Groups’ Comparisons: A Study on PISA Student Questionnaire. Hacettepe University Journal of Education, 30 (4), 80-90
3. Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley- Interscience Publication.
4. Byrne, B. M. (2004). Testing for multigroup invariance using AMOS graphics: A road less traveled. Structural equation modeling, 11(2), 272-300.
5. Bryne, B. M. & Watkins, D. (2003). The issue of measurement invariance revisited. Journal Of Cross-Cultural Psychology, 34, (2), 155-175.
6. Brown, T.A. (2015). Confirmatory factor analysis for applied research. (Second Edition). New York, The Guilford Press.
7. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-Fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233-255.
8. Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk, Ş. (2018). Sosyal bilimler için çok değişkenli istatistik: SPSS ve LISREL uygulamaları. Ankara, Pegem.

9. Demir, Ö., & Seferoğlu, S.S. (2017). İşbirlikli Problem Çözmenin Kodlama Eğitimine Yansıması Olarak Eşli Kodlamanın İncelenmesi. 5th International Instructional Technologies& Teacher Education Symposium. 11-13 October 2017, İzmir.
10. Erkoç, M.F. (2018). The effect of collaborative game design on critical thinking, problem solving and algorithm development skills. Doctoral Dissertation, İstanbul University, İstanbul.
11. Greiff, S., Wüstenberg, S., Molnár, G., Fischer, A., Funke, J., & Csapó, B. (2013). Complex problem solving in educational contexts - something beyond g: Concept, assessment, measurement invariance, and construct validity. Journal of Educational Psychology, 105(2), 364-379.
12. Hesse, F. W. (2017). Designs for operationalizing collaborative problem solving for automated assesment. Journal of Educational Measurement, Spring, 54(1), 12-35.
13. Jöreskog, K. G., & Sörbom, D. (1993). Lisrel 8: Structural equation modeling with the simplis command language. Lincolnwood: Scientific Software International, Inc.
14. Karakoç Alatlı, B. (2016). Investigation of measurement invariance of literacy tests in the Programme for International Student Assessment (PISA - 2012). Published Doctoral Dissertation, Ankara University, Ankara.
15. Kıbrıslıoğlu, N. (2015). The investigation of measurement invariance PISA 2012 mathematics learning model according to culture and gender: Turkey - China (Shangai) – Indonesia. Master Thesis, Hacettepe University, Ankara.
16. Lance, C. E. & Vandenberg, R. J., (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational research methods, 3(1), 4-70.
17. Mark, B. A., & Wan, T.T.H (2005). Testing measurement equivalence in a patient satisfaction instrument. Western Journal of Nursing Research, 27(6), 772-787.
18. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Pyschometrika, 58, 525-543.
19. Millsap, R. E., & Kwok, O. M. (2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological methods, 9(1), 93.
20. Nelson, L.M. (2009). Instructional-design theories and models: A new paradigm of instructional theory, Volume II. New York: Routledge Publishing, (ISBN 978-0-8058-2859-7)
21. OECD, 2017. PISA 2015 results (Volume 5): Collaborative problem solving. Paris: OECD Publishing.
22. O'Neil, H. F., Chuang, S. H., & Chung, G. K. (2003). Issues in the computer-based assessment of collaborative problem solving. Assessment in Education: Principles, Policy & Practice, 10(3), 361-373.
23. Oliden, P., E., & Lizaso, J, M. (2013) Invariance levels across language versions of the PISA 2009 reading comprehension tests in Spain, Psicothema, 25(3), 390-395.
24. Özdemir, S. (2005). The effects of individual and collaborative problem-based learning using an online asynchronized learning tool on critical thinking abilities, academic achievements, and attitudes toward internet use. Doctoral Dissertation, Gazi University, Ankara.
25. Raju, N. S., Laffitte, L. J., & Byrne, B. M. (2002). Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology, 87(3), 517.
26. Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of psychological research online, 8(2), 23-74.
27. Tabachnick, B. G., Fidell, L. S., & Ullman, J. B. (2007). Using multivariate statistics (Vol. 5). Boston, MA: Pearson.
28. Uyar, Ş. (2011). An investigation of measurement invariance of learning strategies model across different groups in PISA Turkey sample. Master Thesis, Hacettepe University, Ankara.
29. Uzunosmanoğlu, S.D. (2013). Examining computer supported collaborative problem solving processes using the dual- eye tracking paradigm. Master’s Thesis, Orta Doğu Teknik Üniversitesi, Ankara.
30. Wang, J & Wang, X. (2012). Structural equation modelling: Applications using mplus. (First edition). UK: Wiley Publications.
31. Wu, D. A., Li, Z., & Zumbo, B. D. (2007). Decoding the meaning of factorial ınvariance and updating the practice of multi-group confirmatory factor analysis: A demonstration with TIMSS data. Practical Assesment, Research and Evaluation, 12(3), 1-26.
32. Yandı, A., Köse, İ. A., & Uysal, Ö. (2017). Farklı Yöntemlerle Ölçme Değişmezliğinin İncelenmesi: Pisa 2012 Örneği. Mersin University Journal of the Faculty of Education, 13(1), 243-253.