Assessment of Mathematical Reasoning Competence in Accordance with PISA 2021 Mathematics Framework

Abstract

The aim of this study is to develop a rubric to evaluate mathematical reasoning proficiency. Within the scope of this purpose, each of the actions expected from the reasoning process in the PISA 2021 Mathematics Framework was taken as criteria by the researchers and a rubric was created. Data were collected from 30 primary school mathematics teachers with the data collection tool containing reasoning questions prepared by the researchers and these data were analyzed using the finalized rubric. In reliability analysis; The "percentage of agreement" among researchers was used to determine the criteria included in the rubric and to be followed with the data collection tool, "Cohen's kappa coefficient" and "Krippendorff's alpha coefficient" methods were used for the agreement between the raters after the analysis of the collected data. The findings show that the Reasoning Competence Rubric (RCR), which consists of 12 criteria, is valid and reliable at a sufficient level.

Keywords

• Mathematical Reasoning
• Competence
• Mathematical Reasoning Rubric
• Mathematical Literacy
• PISA

References

1. Altıparmak, K., & Turgut, Ö. (2005). An Investigation Upon Mathematical Proof and Development of Mathematical Reasoning. Ege Journal of Education, 6(1), 25-37.
2. Altun, M. (2020). Matematik okuryazarlığı el kitabı. Bursa Alfa Aktüel Akademi Yayıncılık.
3. Australian Curriculum & Assessment Authority (ACARA) (2017) Australian Curriculum: Mathematics. Retrieved from https://www.australiancurriculum.edu.au/f-10-curriculum/mathematics/rationale/
4. Bal İncebacak, B., & Ersoy, E. (2016). Analysis Of Mathematical Reasoning Ability Of The Grade 7 Students According To TIMSS. Journal of International Social Research, 9(46), 474-481.
5. Ball, D. L. & Bass, H. (2003). Making Mathematics Reasonable in School. J. Kilpatrick, W. G. Martin. & D. Schifter. (Eds.), A research companion to principles and standards for school mathematics (pp 227-236). Reston, VA: National Council of Teachers of Mathematics.
6. Bıkmaz Bilgen, Ö., & Doğan, N. (2017). The Comparison of Interrater Reliability Estimating Techniques. Journal of Measurement and Evaluation in Education and Psychology, 8(1), 63-78. https://doi.org/10.21031/epod.294847
7. Brualdi Timmins, A. C. (1998). Implementing performance assessment in the classroom. Practical Assessment, Research, and Evaluation, 6(1). https://doi.org/10.7275/kgwx-6q70
8. Bülbül, B.Ö. (2019). How to create and use rubrics? S. Çepni & Y. Kara (Ed.), Current studies in educational measurement evaluation (ss. 257-268). Çanakkale: Paradigma Akademi.

9. Bülbül, S., & Bülbül, B. Ö. (2021). Assessment of Secondary School Textbooks: A Rubric Development Study. YYU Journal of Education Faculty, 18(1), 882-911. https://doi.org/10.33711/yyuefd.938727
10. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104
11. Çoban, H., & Tezci, E. (2020). Developing The Mathematical Reasoning Skills Assessment Scale. OPUS International Journal of Society Researches, 15(24), 2805-2837. https://doi.org/10.26466/opus.610197
12. Dabic Boricic, M., Vulic, I., & Videnovic, M. (2020). Mathematical literacy and assessment: Differences between the PISA study paradigm and mathematics teachers’ conceptions. Malta Review of Educational Research, 14(S), 101-121.
13. Demir, G., & Vural, R. A. (2016). The Examination of Teacher Opinions on Mathematical Competence and Skills of Secondary Mathematics Curriculum. Adnan Menderes University, Journal of Institute of Social Sciences, 4(1), 118-139. https://doi.org/10.30803/adusobed.309074
14. Erdem, E. (2015). The effect of enriched learning environment on mathematical reasoning and attitude (Publication No. 381651) [Doctoral dissertation, Atatürk University]. YÖK. https://tez.yok.gov.tr/
15. Ersanlı, E., Gencoglu, C., & Duran, V. (2018). A Meta-Synthesis Study on Investigations of Dissertations on Education Regarding Reasoning Skills in Turkey. International Online Journal of Educational Sciences, 10(5), 222-243. https://doi.org/10.15345/iojes.2018.05.015
16. Freelon, D. G. (2010). ReCal: Intercoder reliability calculation as a web service. International Journal of Internet Science, 5(1), 20-33. http://dfreelon.org/utils/recalfront/recal2/
17. Gatabi, A. R., Stacey, K., & Gooya, Z. (2012). Investigating grade nine textbook problems for characteristics related to mathematical literacy. Mathematics Education Research Journal, 24(4), 403-421. https://doi.org/10.1007/s13394-012-0052-5
18. Goodrich Andrade, H. (1997). Understanding rubrics. Educational Leadership, 54(4), 14-17.
19. Güler, N. (2019). Eğitimde ölçme ve değerlendirme (14.Baskı). Pegem Akademi. doi: 10.14527/9786053641247
20. Güzel, S. (2017). The evaluation and development of maths lesson teaching program at 6th grades from the proficiencies of mathematical literacy (Publication No. 501514) [Master's thesis, Uludağ University]. YÖK. https://tez.yok.gov.tr/
21. Herbert, S. (2019). Challenges in Assessing Mathematical Reasoning. Mathematics Education Research Group of Australasia, 348-355. https://files.eric.ed.gov/fulltext/ED604186.pdf
22. Herman, T. (2018). Analysis of students’ mathematical reasoning. In Journal of Physics: Conference Series (Vol. 948, No. 1, p. 012036). IOP Publishing. https://doi.org/10.1088/1742-6596/948/1/012036
23. İskenderoğlu, T., & Baki, A. (2011). Classification of the Questions in an 8th Grade Mathematics Textbook with Respect to the Competency Levels of PISA. Education and Science, 36(161), 287-301.
24. Kanik, E. A., Temel, G. O., & Kaya, I. E. (2010). Effect of Sample Size, The Number of Raters and the Category Levels of Diagnostic Test on Krippendorff Alpha and the Fleiss Kappa Statistics for Calculating Inter Rater Agreement: A Simulation Study. Türkiye Klinikleri Biyoistatistik J Biostat, 2(2), 74-81.
25. Kilpatrick, J., & Swafford, J. and B. Findell (Ed.). (2002). Adding it up: Helping children learn mathematics. National Academy Press.
26. Krippendorff, K. (1995). On the reliability of unitizing continuous data. Sociological Methodology, 25, 47-76. https://doi.org/10.2307/271061
27. Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159-174. https://doi.org/10.2307/2529310
28. Leighton, J. P. (2003). Defining and Describing Reasoning. J. P. Leighton & R. J. Sternberg (Ed.), The Nature of Reasoning (ss. 3-11). New York, NY: Cambridge.
29. Loong, E., Vale, C., Widjaja, W., Herbert, S., Bragg, L. A., & Davidson, A. (2018). Developing a Rubric for Assessing Mathematical Reasoning: A Design-Based Research Study in Primary Classrooms. Hunter, J., Perger, P., & Darragh, L. (Ed.), Making waves, opening spaces (ss. 503- 510). Auckland: MERGA.
30. Matteson, S. M. (2006). Mathematical literacy and standardized mathematical assessments. Reading psychology, 27(2-3), 205-233. https://doi.org/10.1080/02702710600642491
31. Ministry of National Education [MoNE]. (2006). PISA 2006 Turkey national final report. Ministry of National Education, Education Research and Development Department, Ankara.
32. Ministry of National Education [MoNE]. (2013). Secondary school mathematics curriculum(5th, 6th, 7th and 8th grades). Ministry of National Education, Board of Education and Discipline, Ankara.
33. Miles, M. B. & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. California, Sage Publications.
34. Moskal, B. M. & Leydens, J. A. (2000). Scoring rubric development: Validity and reliability. Practical Assessment, Research & Evaluation, 7(10), 1-6. https://doi.org/10.7275/q7rm-gg74
35. Moskal, B. M. (2000). Scoring rubrics: What, when and how? Practical Assessment Research and Evaluation,7(3). https://doi.org/10.7275/a5vq-7q66
36. New Jersey Mathematics Coalition and the New Jersey Department of Education (1996). New Jersey Mathematics Curriculum Framework: The first four standards, standard 4- reasoning, K-12 overview. State of New Jersey Department of Education.
37. Niss, M. (2003). Mathematical competencies and the learning of mathematics: The Danish KOM project. In 3rd Mediterranean conference on mathematical education (ss. 115-124).
38. Niss, M., & Højgaard, T. (2019). Mathematical competencies revisited. Educational Studies in Mathematics, 102(1), 9-28. https://doi.org/10.1007/s10649-019-09903-9
39. OECD (2003). The PISA 2003 assesment framework – mathematics, reading, science and problem solving knowledge and skills. OECD Publishing, Paris.
40. OECD (2013). PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. OECD Publishing. http://dx.doi.org/10.1787/9789264190511-en
41. OECD (2017). PISA 2015 assessment and analytical framework: Science, reading, mathematic, financial literacy and collaborative problem solving. OECD Publishing, Paris. http://dx.doi.org/10.1787/9789264281820-en
42. OECD (2019). PISA 2018 assessment and analytical framework. OECD Publishing, Paris. https://doi.org/10.1787/b25efab8-en.
43. Özgen, K. (2021). Checklist on question design for mathematical literacy. Pegem Journal of Education and Instruction, 11(1), 259-298. https://doi.org/10.14527/pegegog.2021.008
44. Peresini, D. & Webb, N. (1999). Analyzing mathematical reasoning in students‘responses across multiple performance assessment tasks. L. V. Stiff and F. R. Curcio (Ed.), Developing Mathematical Reasoning in Grades K-12 (ss. 156-175). Reston, VA: National Council of Teachers of Mathematics.
45. PISA (2021a). PISA 2021 Mathematics Framework Draft. https://pisa2022-maths.oecd.org/files/PISA%202022%20Mathematics%20Framework%20Draft.pdf
46. PISA (2021b). PISA 2021 Mathematics: A Broadened Perspective https://one.oecd.org/document/EDU/PISA/GB(2017)17/en/pdf
47. Russell, M. K. & Airasian, P. W. (2001). Classroom assessment: Concepts and applications. Boston: McGraw-Hill.
48. Russell, S. J. (1999). Mathematical reasoning in the middle grades. L. V. Stiff and F. R. Curcio (Ed.), Developing mathematical reasoning in grades K-12 (ss. 1–12). Reston, VA: National Council of Teachers of Mathematics.
49. Steen, L. A., Turner, R., & Burkhardt, H. (2007). Developing mathematical literacy. Blum, W., Galbraith, P. L., Henn, H. W., & Niss, M. (Ed.), Modelling and applications in mathematics education (ss. 285-294). Springer, Boston, MA.
50. Steen, L.A. (1999) Twenty questions about mathematical reasoning. Stiff L. (Ed.), Developing mathematical reasoning in grades K-12 (ss. 270-285). National Council of Teachers of Mathematics, Reston, Virginia.
51. Şencan, H. (2005). Sosyal ve Davranışsal Ölçümlerde Güvenilirlik ve Geçerlilik. Ankara, Seçkin Yayıncılık.
52. Toulmin, S., Rieke, R., & Janik, A. (Ed.). (1984). An introduction to reasoning: Second edition. Collier Macmillan Publishers.
53. Venkat, H., Graven, M., Lampen, E., Nalube, P., & Chitera, N. (2009). 'Reasoning and reflecting'in mathematical literacy. Learning and Teaching Mathematics, 2009(7), 47-53.
54. Yackel, E. & Hanna, G. (2003). Reasoning and proof. J. Kilpatrick, G. Martin and D. Schifter (Ed.), A research companion to principles and standards for school mathematics (ss. 227–236). National Council of Teachers of Mathematics.
55. Yankelewitz, D. (2009). The development of mathematical reasoning in elementary school students' exploration of fraction ideas. Rutgers The State University of New Jersey-New Brunswick.
56. Yıldız, H. (2019). Investigation of the challenges of the seventh grade students encountered the solution of mathematical literacy questions. [Unpublished master's dissertation]. Bursa Uludag University.