Identifying the presence of context and item-writing flaws in practice items: The case of Turkish mathematics textbooks

Abstract

This study seeks to ascertain the degree to which context-based items are offered in Turkish mathematics textbooks as well as the quality of the items in terms of item writing guidelines, whether or not they are given as traditional or context-based. A qualitative research approach is used in this study. The eighth-grade mathematics textbook used in public schools and a textbook used in certain private school chains constitute its sample. The practice items (i.e, exercises without solutions given) included in the textbooks were analyzed by performing document analysis. The results revealed that both textbooks contain several flawed items in terms of item writing rules, as well as having mainly non-contextual items.

Keywords

• Context
• Item writing rules
• Mathematics textbooks

Project Number

T-20-B2010-90025

References

1. Başaran, S. (2005). Diğer ülkelerde lise bitirme sınavları ve Türk eğitim sistemi için lise bitirme sınavı önerisi [High school leaving exams in other countries and high school leaving exam recommendation for the Turkish education system]. MEB Eğitimi Araştırma ve Geliştirme Dairesi Başkanlığı.
2. Bolstad, O.H. (2020). Secondary teachers’ operationalisation of mathematical literacy. European Journal of Science and Mathematics Education, 8(3), 115 135. https://doi.org/10.30935/scimath/9551
3. Bowen, G.A. (2009). Document analysis as a qualitative research method. Qualitative Research Journal, 9(2), 27-40. https://doi.org/10.3316/QRJ0902027
4. Burton, S.J., Sudweeks, R.R., Merrill, P.F., & Wood, B. (1991). How to prepare better multiple-choice test items: Guidelines for university faculty. Bringham Young University Testing Services and the Department of Instructional Science. http://testing.byu.edu/info/handbooks/betteritems.pdf
5. Chiavaroli, N. (2017). Negatively-worded multiple choice questions: An avoidable threat to validity. Practical Assessment, Research & Evaluation, 22(3), 1 14. https://doi.org/10.7275/ca7y-mm27
6. Downing, S.M. (2005). The effects of violating standard item writing principles on tests and students: The consequences of using flawed test items on achievement examinations in medical education. Advances in Health Sciences, 10, 133 143. https://doi.org/10.1007/s10459-004-4019-5
7. Fidan, M. (2018). Ortaokul öğrencilerinin Türkçe ders kitaplarının tasarımına yönelik görüşlerinin analizi [Analysis of middle school students' views on the design of Turkish textbooks.]. Bayterek International Journal of Academic Research,1(2), 178–189.
8. Fraenkel, J.R., Wallen, N.E., & Hyun, H.H. (2012). How to design and evaluate research in education (8th ed.). McGraw Hill.

9. Geiger, V., Goos, M., & Forgasz, H. (2015). A rich interpretation of numeracy for the 21st century: A survey of the state of the field. ZDM Mathematics Education, 47(4), 531–548. https://doi.org/10.1007/s11858-015-0708-1
10. Gierl, M.J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87(6), 1082–1116. https://doi.org/10.3102/0034654317726529
11. Goos, M., Geiger, V., & Dole, S. (2012). Auditing the numeracy demands of the middle years curriculum. PNA, 6(4), 147-158. https://doi.org/10.30827/pna.v6i4.6138
12. Güler, H.K. & Ülger, B. (2018). PISA, TIMSS ve TEOG sınavlarının temele aldığı öğrenme kuramları [Learning theories based on PISA, TIMSS and TEOG exams]. In S. Çepni (Ed.), PISA ve TIMSS mantığını ve sorularını anlama (ss.111-153). Pegem A Yayıncılık.
13. Hadar, L.L.(2017). Opportunities to learn: Mathematics textbooks and students’achievements, Studies in Educational Evaluation, 55, 153 166. http://dx.doi.org/10.1016/j.stueduc.2017.10.002
14. Kahraman, İ. (2014). Merkezi ortak sınav uygulamasının etkilerine ilişkin öğretmen görüşleri [The effect of common implementation that related to teachers’ opinion]. Tunceli Üniversitesi Sosyal Bilimler Dergisi, 2(4), 53-74.
15. Kaiser, G., & Willander, T. (2005). Development of mathematical literacy: results of an empirical study. Teaching Mathematics and Its Applications, 24(2-3), 48-60. https://doi.org/10.1093/teamat/hri016
16. Kar, T. & Işık, C. (2015). Comparison of Turkish and American seventh grade mathematics textbooks in terms of addition and subtraction operations with integers. Education and Science, 40(177), 75-92. https://doi.org/10.15390/EB.2015.2897
17. Kayhan Altay, M., Kurt Erhan, G. & Batı, E. (2020). Contexts used for real life connections in mathematics textbook for 6th graders. Elementary Education Online, 19(1), 310-323. https://doi.org/10.17051/ilkonline.2020.656880
18. Korkmaz, E., Tutak, T., & İlhan, A. (2020). Ortaokul matematik ders kitaplarının matematik öğretmenleri tarafından değerlendirilmesi [Evaluation of secondary school mathematics textbooks by mathematics teachers]. Avrupa Bilim ve Teknoloji Dergisi, 18, 118-128. https://doi.org/10.31590/ejosat.667689
19. Krouse, S. (2016, Jan 8). Why do we need to learn this. Medium. https://medium.com/@stevekrouse/why-do-we-need-to-learn-this-3ba1d42bd08a.
20. Kul, Ü., Sevimli, E., & Aksu, Z. (2018). A comparison of mathematics questions in Turkish and Canadian school textbooks in terms of synthesized taxonomy. Turkish Journal of Education, 7(3), 136-155. https://doi.org/10.19128/turje.395162
21. Miles, M.B., & Huberman, A.M. (1994). Qualitative data analysis: An expanded sourcebook. Sage.
22. Miller, D.M., Linn, R.L, & Gronlund, N.E. (2013). Measurement and assessment in teaching (11th ed.). Pearson Education, Inc.
23. MoNE (Ministry of National Education) (2019). PISA 2018 ulusal ön raporu [PISA 2018 Preliminary National Report]. Eğitim Analiz ve Değerlendirme Raporları Serisi,10.
24. OECD (2009). Learning mathematics for life: A view perspective from PISA OECD Publishing.
25. OECD (2019a). PISA 2018 results (Volume I): What students know and can do. PISA OECD Publishing. https://doi.org/10.1787/5f07c754-en
26. OECD, (2019b). PISA 2018 assessment and analytical framework. PISA OECD Publishing. https://doi.org/10.1787/b25efab8-en
27. Osterlind, S.J. (2002). What is constructing test items? In S. J. Osterlind (Ed.), Constructing test items: Multiple-choice, constructed-response, performance, and other formats (pp. 1–16), Springer. https://doi.org/10.1007/0-306-47535-9_1
28. Peeters, M.J., Beltyukova, S.A., & Martin, B.A. (2013). Educational testing and validity of conclusions in the scholarship of teaching and learning. American Journal of Pharmaceutical Education, 77(9), 1-9. https://doi.org/10.5688/ajpe779186
29. Rahimah, D. & Visnovska, J. (2021). Analysis of mathematics textbook use: An argument for combining horizontal, vertical, and contextual analyses. Journal of Physics: Conference Series, 1731, 1-5. https://doi.org/10.1088/1742-6596/1731/1/01204
30. Rea-Dickson P. & Germania, K. (2001). Evaluating curriculum change. In D. Hall & A. Hewimng (Eds.), Innovation in English language teaching: A reader. British Library Catalogue.
31. Royal, K.D. & Stockdale, M.R. (2017). The impact of 3-option responses to multiple-choice questions on guessing strategies and cut score determinations. Journal of Advances in Medical Education & Professionalism, 5(2), 84-89.
32. Rush, B.R., Rankin, D.C & White, B.J. (2016). The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Medical Education, 16(250), 1-10. https://doi.org/10.1186/s12909-016-0773-3
33. Schwarzkopf, R. (2007). Elementary modeling in mathematics lessons: The interplay between real-world knowledge and mathematics structures. In W. Blum, P. L. Galbraith, H.W. Henn, & M. Niss (Eds.), Modelling and applications in mathematics education: The 14th ICMI study (pp. 209–216). Springer.
34. Shin, J., Guo, Q., & Gierl, M. J. (2019). Multiple-choice item distractor development using topic modeling approaches. Frontiers in Psychology, 10, 1 14. https://doi.org/10.3389/fpsyg.2019.00825
35. Simsek, A. (2016). A comparative analysis of common mistakes in achievement tests prepared by school teachers and corporate trainers. European Journal of Science and Mathematics Education, 4(4), 477-489.
36. Törnroos, J. (2005). Mathematics textbooks, opportunity to learn and student achievement. Studies in Educational Evaluation, 31, 315 327. https://doi.org/10.1016/j.stueduc.2005.11.005
37. Valverde, G., Bianchi, L, Wolfe, R., Schmidt, W. & Houang, R. (2002). According to the book: Using TIMSS to investigate the translation of policy into practice through the world of textbooks. Kluwer Academic Publishers.
38. Van den Heuvel-Panhuizen, M. (2005). The role of context in assessment problems in mathematics. For the Learning of Mathematics, 25(2), 2-23.
39. Verschaffel, L., Greer, B., & De Corte, E. (2000). Making sense of word problems. Swets & Zeitlinger.
40. Wijaya, A., van den Heuvel-Panhuizen, M., & Doorman, M. (2015). Opportunity-to-learn context-based tasks provided by mathematics textbooks. Educational Studies in Mathematics, 89(1), 41-65. https://doi.org/10.1007/s10649-015-9595-1
41. Yam, H. (2005). What is contextual learning and teaching in physics? Retrieved from http://www.phy.cuhk.edu.hk/contextual/approach/tem/brief_e.html