An Examination of Turkish Students' PISA 2015 Collaborative Problem-Solving Competencies

Abstract

Technological advancements initially made it possible for individuals with different cultures from different parts of the world to consider working together, and eventually collaboration turned into a necessity. Based on this necessity, OECD measured collaborative problem-solving (CPS) competencies of 15-year-old students from the countries participating in PISA assessment in 2015. The objective of this study is to examine the data on Turkey from PISA 2015 CPS survey through Deterministic-Input, Noisy-Or-Gate (DINO) cognitive diagnostic model and to determine the students’ levels of CPS competencies as well as difficulty levels of these competencies. In this context, primarily, attributes of CPS competency domain were examined and the model with the most appropriate number of attributes, classification accuracy and consistency was determined for the analysis of PISA 2015 CPS data. The sample of this descriptive study consists of 435 students. As a result of the analysis, the Q-matrix of the model data fit was observed to have the best goodness-of-fit when it had three attributes whereas model fit and classification consistency decreased as the number of attributes increased. Furthermore, the easiest competency for Turkish students was determined to be ‘taking appropriate action to solve the problem’ while the most difficult was ‘establishing and maintaining shared understanding’. Finally, latent class distribution exposed that 56% of the students did not have any of the competencies, and 35% had all. Results of this examination indicate that all CPS competencies of Turkish students need improvement.

Keywords

• PISA 2015
• Turkey
• Collaborative Problem-Solving
• Cognitive Diagnostic Model
• DINO

An Examination of Turkish Students' PISA 2015 Collaborative Problem-Solving Competencies

Öz

Technological advancements initially made it possible for individuals with different cultures from different parts of the world to consider working together, and eventually collaboration turned into a necessity. Based on this necessity, OECD measured collaborative problem-solving (CPS) competencies of 15-year-old students from the countries participating in PISA assessment in 2015. The objective of this study is to examine the data on Turkey from PISA 2015 CPS survey through Deterministic-Input, Noisy-Or-Gate (DINO) cognitive diagnostic model and to determine the students’ levels of CPS competencies as well as difficulty levels of these competencies. In this context, primarily, attributes of CPS competency domain were examined and the model with the most appropriate number of attributes, classification accuracy and consistency was determined for the analysis of PISA 2015 CPS data. The sample of this descriptive study consists of 435 students. As a result of the analysis, the Q-matrix of the model data fit was observed to have the best goodness-of-fit when it had three attributes whereas model fit and classification consistency decreased as the number of attributes increased. Furthermore, the easiest competency for Turkish students was determined to be ‘taking appropriate action to solve the problem’ while the most difficult was ‘establishing and maintaining shared understanding’. Finally, latent class distribution exposed that 56% of the students did not have any of the competencies, and 35% had all. Results of this examination indicate that all CPS competencies of Turkish students need improvement.

Anahtar Kelimeler

• PISA 2015
• Collaborative ProblemSolving
• Cognitive Diagnostic Model
• DINO

References

1. Altun, M. (2000). İlköğretimde problem çözme öğretimi [Teaching problem solving in primary education], Journal of Education and Social Sciences, 147. Retrieved July 10, 2019, from http://dhgm.meb.gov.tr/yayimlar/dergiler/Milli_Egitim_Dergisi/147/altun.htm
2. Altun, M. (2014). Eğitim Fakülteleri ve İlköğretim Matematik Öğretmenleri için Matematik Öğretimi [Teaching Mathematics for Faculty of Education and Primary School Mathematics Teachers], 18th ed.; Bursa: Alfa Aktüel Publication.
3. Arıcı, Ö. (2019). Investigating the factors related to Turkish students' collaborative problem solving skills with mediating models according to PISA 2015 results. Doctoral Dissertation, University of Ankara at Ankara.
4. Arslan, Ç., Altun, M. (2007). Learning to solve non-rou¬tine problems. Elementary Education Online, 6(1), 50-61.
5. Baki, A. (2006). Kuramdan Uygulamaya Matematik Eğitimi [Mathematics Education from Theory to Practice]. İstanbul: Bilge Printing.
6. Bozkurt, Ş. B., Çakır, H. (2016). 21st Century learner skills: An investigation of middle school students based on grade level and gender. Pamukkale University Journal of Education, 39, 69-82.
7. Chang, C.-J., Chang, M.-H., Chiu, B.-C., Liu, C.-C., Chiang, S.-H. F., Wen, C.-T., et al. (2017) An analysis of student collaborative problem solving activities mediated by collaborative simulations. Computers & Education, 114, 222 235. https://doi.org/10.1016/j.compedu.2017.07.008
8. Cui, Y., Gierl, M. J., Chang, H. (2012). Estimating classification consistency and accuracy for cognitive diagnostic assessment. Journal of Educational Measurement, 49(1), 19–38. https://doi.org/10.1111/j.1745-3984.2011.00158.x

9. Cui, Y., Gierl, M., Guo, Q. (2015). Statistical classification for cognitive diagnostic assessment: An artificial neural network approach. Educational Psychology: An International Journal of Experimental Educational Psychology, 1 24. https://doi.org/10.1080/01443410.2015.1062078
10. Dede Y., Yaman S. (2006). Fen ve matematik eğitiminde problem çözme: Kuramsal bir çalışma [Problem solving in science and mathematics education: A theoretical study]. Çukurova University Faculty of Education Journal, 2, 116-128.
11. Dewey, J. (1933). How We Think: A Restatement of the Relation of Reflective Thinking to The Educative Process. Boston: DC Heath and Company.
12. DiBello, L. V., Roussos, L. A., Stout, W. F. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. Handbook of Statistics, 26, 979–1030. https://doi.org/10.1016/S0169-7161(06)26031-0
13. Dillenbourg, P., Baker, M., Blaye, A., O’Malley, C. (1996). The evolution of research on collaborative learning. In E. Spada, P. Reiman (Eds.), Learning in Humans and Machines: Towards an Interdisciplinary Learning Science (pp. 189–211). Oxford: Elsevier.
14. Frederiksen, N. (1984). Implications of cognitive theory for instruction in problem solving. Review of Educational Research 54(3), 363–407. https://doi.org/10.1002/j.2330-8516.1983.tb00019.x
15. Gök, T., Sılay, İ.(2008). The effects of problem-solving strategies on students’ achievement, achievement motivation and attitude in the cooperative learning groups in physics teaching. Hacettepe University Journal of Education, 34, 116-126.
16. Gök, T., Sılay, İ. (2009). The effects of problem-solving strategies teaching on students’ achievement motivation, in the cooperative learning groups. Kastamonu Education Journal, 17(3), 821-834.
17. Hartman, H. J. (1998). Metacognition in teaching and learning: An introduction. Instructional Science, 26, 1-3. https://doi.org/10.1023/A:1003023628307
18. Herborn, K., Stadler, M., Mustafić, M., Greiff, S. (2018). The assessment of collaborative problem solving in PISA 2015: Can computer agents replace humans? Computers in Human Behavior, 87, 1-31. https://doi.org/10.1016/j.chb.2018.07.035
19. Hesse, F., Care, E., Buder, J., Sassenberg, K., Griffin, P. (2015). A framework for teachable collaborative problem-solving skills. In P. Griffin, E. Care (Eds.), Assessment and Teaching of 21st Century Skills: Methods and Approach (pp. 37-56). Dordrecht: Springer.
20. Hmelo C., E. Silver (2004). Problem based learning; what and how do students learn? Educational Psychology Review, 16(39), 235 263. https://doi.org/10.1023/B:EDPR.0000034022.16470.f3
21. IBM Corp. (2013). IBM SPSS Statistics for Windows. Version 22.0. Armonk, NY: IBM Corp.
22. Kneeland, S. (1999). Effective Problem-Solving: How to Understand the Process and Practise it Successfully. London: Little, Brown Book Group.
23. Krulik, S., Rudnick, J. A. (1985). Developing problem solving skills. Mathematics Teacher, 79(9), 685-692.
24. Kunina-Habenicht, O., Rupp, A. A., Wilhelm, O. (2009). A practical illustration of multidimensional diagnostic skills profiling: Comparing results from confirmatory factor analysis and diagnostic classification models. Studies in Educational Evaluation, 35, 64–70. https://doi.org/10.1016/j.stueduc.2009.10.003
25. Kuo, B.-C., Liao, C.-H., Pai, K.-C., Shih, S.-C., Li, C.-H., Mok, M. M. C. (2019). Computer-based collaborative problem-solving assessment in Taiwan. Educational Psychology, 39(1), 1-22. https://doi.org/10.1080/01443410.2018.1549317
26. Kutluca, T. (2013). The effect of geometry instruction with dynamic geometry software; GeoGebra on Van Hiele geometry understanding levels of students. Educational Research and Reviews, 8(17), 1509-1518. https://doi.org/10.5897/ERR2013.1554
27. Lesh, R., Zawojewski, J.S. (2007). Problem solving and modeling. In F. Lester (Ed.), Second Handbook of Research on Mathematics Teaching and Learning (pp. 763-804). USA: Information Age Publishing.
28. Li, C.-H., Liu, Z.-Y. (2017). Collaborative problem-solving behavior of 15-year-old taiwanese students in science education. EURASIA Journal of Mathematics Science and Technology Education, 13(10), 6677-6695. https://doi.org/10.12973/ejmste/78189 Lin, K‑Y., Yu, K‑C., Hsiao, H‑S., Chang, Y‑S., & Chien, Y‑H. (2018). Effects of web‑based versus classroom‑based STEM learning environments on the development of collaborative problem‑solving skills in junior high school students. International Journal of Technology and Design Education, 1-14. https://doi.org/10.1007/s10798-018-9488-6
29. Madison, M. J., Bradshaw, L. P. (2015). The effects of Q-matrix design on classification accuracy in the log-linear cognitive diagnosis model. Educational and Psychological Measurement, 75(3) 491–511. https://doi.org/10.1177/0013164414539162
30. Mayer, R.E. (1990). Problem solving. In M. W. Eysenck (Ed.), The Blackwell Dictionary of Cognitive Psychology (pp.284-288). Oxford: Basil Blackwell.
31. Mayer, R. E., Hegarty, M. (1996). The process of understanding mathematical problems. In R.J. Sternberg, T. Ben-Zeev (Eds.), The Nature of Mathematical Thinking (pp.29-54). Hillsdale, NJ: Lawrence Erlbaum Associates.
32. Ministry of Education (2009). İlköğretim Matematik Dersi 6-8. Sınıflar Öğretim Programı ve Klavuzu [Elementary Mathematics 6-8th Classes Curriculum and Guide]. Ankara. Retrieved Jun 10, 2019, from http://mimoza.marmara.edu.tr/~apusmaz/dosyalar/6-8_Program%C4%B1.pdf
33. Nancarrow, M. (2004). Exploration of metacognition and non-routine problem based mathematics instruction on undergraduate student problem solving success. Doctoral Dissertation, Florida State University, Florida.
34. National Council of Teachers of Mathematics (NCTM). (1989). Curriculum and Evaluation Standards for School Mathematics. Virginia, Reston: NCTM Inc. Retrieved Jun 10, 2019, from http://csmc.missouri.edu/PDFS/CCM/summaries/standards_summary.pdf
35. Nordin, N.M., Osman, K. (2018). Students’ generated animation: An innovative approach to inculcate collaborative problem solving (CPS) skills in learning physics. Journal of Education in Science, Environment and Health (JESEH), 4(2), 206 226. https://doi.org/10.21891/jeseh.436758
36. OECD, (1999). PISA 2000 Measuring student knowledge and skills: A new framework for assessment, OECD Publishing, Paris. Retrieved May 1, 2019, from https://www.oecd.org/education/school/programmeforinternationalstudentassessmentpisa/33693997.pdf
37. OECD, (2003). The PISA 2003 assessment framework: Mathematics, reading, science and problem solving knowledge and skills, OECD Publishing, Paris. Retrieved May 1, 2019, from https://www.oecd.org/education/school/programmeforinternationalstudentassessmentpisa/33694881.pdf
38. OECD, (2006). Assessing Scientific, Reading and Mathematical Literacy: A framework for PISA 2006, OECD Publishing, Paris. Retrieved May 1, 2019, from https://www.sel-gipes.com/uploads/1/2/3/3/12332890/assessing_scientific_reading_and_mathematical_literacy.pdf
39. OECD, (2009). PISA 2009 assessment framework – Key competencies in reading, mathematics and science, OECD Publishing, Paris. Retrieved May 1, 2019, from https://www.oecd.org/pisa/pisaproducts/44455820.pdf
40. OECD (2011). Skills for Innovation and Research, OECD Publishing, Paris. https://doi.org/10.1787/9789264097490-en
41. OECD (2013). PISA 2012 Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy, OECD Publishing, Paris. https://doi.org/10.1787/9789264190511-en
42. OECD (2017a). PISA 2015 collaborative problem-solving framework, OECD Publishing, Paris. Retrieved May 1, 2019, from https://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Collaborative%20Problem%20Solving%20Framework%20.pdf
43. OECD (2017b). PISA 2015 Results (Volume V): Collaborative Problem Solving, OECD Publishing, Paris. https://doi.org/10.1787/9789264285521-en
44. Özgen, K., Pesen, C. (2010). Probleme dayalı öğrenme (PDÖ) yaklaşımı ile işlenen matematik dersinde öğrencilerin problem çözme becerilerinin analizi [Analysis of students' problem solving skills in mathematics course processed by problem based learning (PBL) approach]. Journal of Education and Social Sciences, 186, 27-37.
45. Özkan, E., Öner D. (2019). Investigation of the development of van Hiele levels of geometric thinking in a computer supported collaborative learning (CSCL) environment. Mersin University Journal of the Faculty of Education, 15(2), 473 490. https://doi.org/10.17860/mersinefd.522491
46. Öztürk, E. (2001). Yaratıcılık ve eğitim [Creativity and education]. Sakarya University Journal of Education Faculty, 1, 158-164.
47. Partnership for 21st Century Skills (2009). P21 Framework definition. Retrieved July 7, 2019, from https://files.eric.ed.gov/fulltext/ED519462.pdf
48. Polya, G. (1973). How to Solve It? USA: Princeton University.
49. Ravand, H. (2016). Application of a cognitive diagnostic model to a high-stakes reading comprehension test. Journal of Psychoeducational Assessment, 34(8), 1-18. https://doi.org/10.1177/0734282915623053
50. Robitzsch, A., Kiefer, T., George, A. C., & Uenlue, A. (2019, September). CDM [software package in R]. Available online at https://cran.r project.org/web/packages/CDM/index.html
51. Rosen, Y., Tager, M. (2013). Computer-based assessment of collaborative problem-solving skills: Human-to-agent versus human-to-human approach. Boston: Pearson Education Retrieved July 7, 2019, from https://www.researchgate.net/publication/258629038_Computer based_assessment_of_collaborative_problem solving_skills_Human to agent_versus_human-to-human_approach_Boston_MA_Pearson_Education
52. Rupp, A. A., Templin, J., Henson, R. A. (2010). Diagnostic Measurement: Theory, Methods, and Applications. New York: The Guilford.
53. Sinharay, S., Johnson, M. (2019). Measures of agreement: Reliability, classification accuracy, and classification consistency. In M. von Davier, Y.-S. Lee (Eds.), Handbook of Diagnostic Classification Models (pp.359-377). New York: Springer.
54. Soylu, Y., Soylu, C. (2006). The role of problem solving in mathematics lessons for success. İnönü University Journal of the Faculty of Education, 7(11), 97-111. Retrieved July 7, 2019, from https://www.pegem.net/dosyalar/dokuman/8298 2011062915226 soylusoylu.pdf
55. Şenşekerci, E., Bilgin, A. (2008). Critical Thinking and its Teaching. Uludağ University Faculty of Arts and Sciences Journal of Social Sciences, 9(14), 15-43.
56. Templin, J., Henson, R. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305. https://doi.org/10.1037/1082-989X.11.3.287
57. Turkish Language Association (2019). Türk Dil Kurumu sözlükleri [Dictionary of Turkish Language Association]. Retrieved July 7, 2019, from http://sozluk.gov.tr/
58. Uzunosmanoğlu, S. D. (2013). Examining computer supported collaborative problem solving processes using the dual-eye tracking paradigm. Master Dissertation, Middle East Technical University at Ankara.
59. Yalçın, S. (2018). 21st century skills and tools and approaches that are used to measure these skills. Ankara University Journal of Faculty of Educational Sciences, 51(1), 183-201.
60. Yavuz, E. (2014). Determining the problem solving process skills of the primary education preservice mathematics teachers as defined in PISA. Master Dissertation, Gazi University at Ankara.
61. Yazlık, D., Ö., Erdoğan, A. (2016). The effects of problem-solving strategies used in combinations with cooperative learning on learner achievement. Ahi Evran University Journal of Kırşehir Education Faculty (KEFAD), 17(3), 1-16.
62. Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145.