int. j. assess. tools educ. International Journal of Assessment Tools in Education 2148-7456 İzzet KARA 10.21449/ijate.1089472 Other Fields of Education Alan Eğitimleri Identifying the presence of context and item-writing flaws in practice items: The case of Turkish mathematics textbooks Identifying the presence of context and item-writing flaws in practice items: The case of Turkish mathematics textbooks https://orcid.org/0000-0002-7098-0118 İlgün Dibek Münevver TED UNIVERSITY, FACULTY OF EDUCATION https://orcid.org/0000-0001-9660-0403 Toker Zerrin TED ÜNİVERSİTESİ, EĞİTİM FAKÜLTESİ 11 29 2022 9 Special Issue 218 235 03 17 2022 11 03 2022 Copyright © 2014, International Journal of Assessment Tools in Education 2014 International Journal of Assessment Tools in Education

This study seeks to ascertain the degree to which context-based items are offered in Turkish mathematics textbooks as well as the quality of the items in terms of item writing guidelines, whether or not they are given as traditional or context-based. A qualitative research approach is used in this study. The eighth-grade mathematics textbook used in public schools and a textbook used in certain private school chains constitute its sample. The practice items (i.e, exercises without solutions given) included in the textbooks were analyzed by performing document analysis. The results revealed that both textbooks contain several flawed items in terms of item writing rules, as well as having mainly non-contextual items.

This study seeks to ascertain the degree to which context-based items are offered in Turkish mathematics textbooks as well as the quality of the items in terms of item writing guidelines, whether or not they are given as traditional or context-based. A qualitative research approach is used in this study. The eighth-grade mathematics textbook used in public schools and a textbook used in certain private school chains constitute its sample. The practice items (i.e, exercises without solutions given) included in the textbooks were analyzed by performing document analysis. The results revealed that both textbooks contain several flawed items in terms of item writing rules, as well as having mainly non-contextual items.

 Context item writing rules mathematics textbook Context Item writing rules Mathematics textbooks TEDU-BAP T-20-B2010-90025 Başaran, S. (2005). Diğer ülkelerde lise bitirme sınavları ve Türk eğitim sistemi için lise bitirme sınavı önerisi [High school leaving exams in other countries and high school leaving exam recommendation for the Turkish education system]. MEB Eğitimi Araştırma ve Geliştirme Dairesi Başkanlığı. Bolstad, O.H. (2020). Secondary teachers’ operationalisation of mathematical literacy. European Journal of Science and Mathematics Education, 8(3), 115 135. https://doi.org/10.30935/scimath/9551 Bowen, G.A. (2009). Document analysis as a qualitative research method. Qualitative Research Journal, 9(2), 27-40. https://doi.org/10.3316/QRJ0902027 Burton, S.J., Sudweeks, R.R., Merrill, P.F., & Wood, B. (1991). How to prepare better multiple-choice test items: Guidelines for university faculty. Bringham Young University Testing Services and the Department of Instructional Science. http://testing.byu.edu/info/handbooks/betteritems.pdf Chiavaroli, N. (2017). Negatively-worded multiple choice questions: An avoidable threat to validity. Practical Assessment, Research & Evaluation, 22(3), 1 14. https://doi.org/10.7275/ca7y-mm27 Downing, S.M. (2005). The effects of violating standard item writing principles on tests and students: The consequences of using flawed test items on achievement examinations in medical education. Advances in Health Sciences, 10, 133 143. https://doi.org/10.1007/s10459-004-4019-5 Fidan, M. (2018). Ortaokul öğrencilerinin Türkçe ders kitaplarının tasarımına yönelik görüşlerinin analizi [Analysis of middle school students' views on the design of Turkish textbooks.]. Bayterek International Journal of Academic Research,1(2), 178–189. Fraenkel, J.R., Wallen, N.E., & Hyun, H.H. (2012). How to design and evaluate research in education (8th ed.). McGraw Hill. Geiger, V., Goos, M., & Forgasz, H. (2015). A rich interpretation of numeracy for the 21st century: A survey of the state of the field. ZDM Mathematics Education, 47(4), 531–548. https://doi.org/10.1007/s11858-015-0708-1 Gierl, M.J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87(6), 1082–1116. https://doi.org/10.3102/0034654317726529 Goos, M., Geiger, V., & Dole, S. (2012). Auditing the numeracy demands of the middle years curriculum. PNA, 6(4), 147-158. https://doi.org/10.30827/pna.v6i4.6138 Güler, H.K. & Ülger, B. (2018). PISA, TIMSS ve TEOG sınavlarının temele aldığı öğrenme kuramları [Learning theories based on PISA, TIMSS and TEOG exams]. In S. Çepni (Ed.), PISA ve TIMSS mantığını ve sorularını anlama (ss.111-153). Pegem A Yayıncılık. Hadar, L.L.(2017). Opportunities to learn: Mathematics textbooks and students’achievements, Studies in Educational Evaluation, 55, 153 166. http://dx.doi.org/10.1016/j.stueduc.2017.10.002 Kahraman, İ. (2014). Merkezi ortak sınav uygulamasının etkilerine ilişkin öğretmen görüşleri [The effect of common implementation that related to teachers’ opinion]. Tunceli Üniversitesi Sosyal Bilimler Dergisi, 2(4), 53-74. Kaiser, G., & Willander, T. (2005). Development of mathematical literacy: results of an empirical study. Teaching Mathematics and Its Applications, 24(2-3), 48-60. https://doi.org/10.1093/teamat/hri016 Kar, T. & Işık, C. (2015). Comparison of Turkish and American seventh grade mathematics textbooks in terms of addition and subtraction operations with integers. Education and Science, 40(177), 75-92. https://doi.org/10.15390/EB.2015.2897 Kayhan Altay, M., Kurt Erhan, G. & Batı, E. (2020). Contexts used for real life connections in mathematics textbook for 6th graders. Elementary Education Online, 19(1), 310-323. https://doi.org/10.17051/ilkonline.2020.656880 Korkmaz, E., Tutak, T., & İlhan, A. (2020). Ortaokul matematik ders kitaplarının matematik öğretmenleri tarafından değerlendirilmesi [Evaluation of secondary school mathematics textbooks by mathematics teachers]. Avrupa Bilim ve Teknoloji Dergisi, 18, 118-128. https://doi.org/10.31590/ejosat.667689 Krouse, S. (2016, Jan 8). Why do we need to learn this. Medium. https://medium.com/@stevekrouse/why-do-we-need-to-learn-this-3ba1d42bd08a. Kul, Ü., Sevimli, E., & Aksu, Z. (2018). A comparison of mathematics questions in Turkish and Canadian school textbooks in terms of synthesized taxonomy. Turkish Journal of Education, 7(3), 136-155. https://doi.org/10.19128/turje.395162 Miles, M.B., & Huberman, A.M. (1994). Qualitative data analysis: An expanded sourcebook. Sage. Miller, D.M., Linn, R.L, & Gronlund, N.E. (2013). Measurement and assessment in teaching (11th ed.). Pearson Education, Inc. MoNE (Ministry of National Education) (2019). PISA 2018 ulusal ön raporu [PISA 2018 Preliminary National Report]. Eğitim Analiz ve Değerlendirme Raporları Serisi,10. OECD (2009). Learning mathematics for life: A view perspective from PISA OECD Publishing. OECD (2019a). PISA 2018 results (Volume I): What students know and can do. PISA OECD Publishing. https://doi.org/10.1787/5f07c754-en OECD, (2019b). PISA 2018 assessment and analytical framework. PISA OECD Publishing. https://doi.org/10.1787/b25efab8-en Osterlind, S.J. (2002). What is constructing test items? In S. J. Osterlind (Ed.), Constructing test items: Multiple-choice, constructed-response, performance, and other formats (pp. 1–16), Springer. https://doi.org/10.1007/0-306-47535-9_1 Peeters, M.J., Beltyukova, S.A., & Martin, B.A. (2013). Educational testing and validity of conclusions in the scholarship of teaching and learning. American Journal of Pharmaceutical Education, 77(9), 1-9. https://doi.org/10.5688/ajpe779186 Rahimah, D. & Visnovska, J. (2021). Analysis of mathematics textbook use: An argument for combining horizontal, vertical, and contextual analyses. Journal of Physics: Conference Series, 1731, 1-5. https://doi.org/10.1088/1742-6596/1731/1/01204 Rea-Dickson P. & Germania, K. (2001). Evaluating curriculum change. In D. Hall & A. Hewimng (Eds.), Innovation in English language teaching: A reader. British Library Catalogue. Royal, K.D. & Stockdale, M.R. (2017). The impact of 3-option responses to multiple-choice questions on guessing strategies and cut score determinations. Journal of Advances in Medical Education & Professionalism, 5(2), 84-89. Rush, B.R., Rankin, D.C & White, B.J. (2016). The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Medical Education, 16(250), 1-10. https://doi.org/10.1186/s12909-016-0773-3 Schwarzkopf, R. (2007). Elementary modeling in mathematics lessons: The interplay between real-world knowledge and mathematics structures. In W. Blum, P. L. Galbraith, H.W. Henn, & M. Niss (Eds.), Modelling and applications in mathematics education: The 14th ICMI study (pp. 209–216). Springer. Shin, J., Guo, Q., & Gierl, M. J. (2019). Multiple-choice item distractor development using topic modeling approaches. Frontiers in Psychology, 10, 1 14. https://doi.org/10.3389/fpsyg.2019.00825 Simsek, A. (2016). A comparative analysis of common mistakes in achievement tests prepared by school teachers and corporate trainers. European Journal of Science and Mathematics Education, 4(4), 477-489. Törnroos, J. (2005). Mathematics textbooks, opportunity to learn and student achievement. Studies in Educational Evaluation, 31, 315 327. https://doi.org/10.1016/j.stueduc.2005.11.005 Valverde, G., Bianchi, L, Wolfe, R., Schmidt, W. & Houang, R. (2002). According to the book: Using TIMSS to investigate the translation of policy into practice through the world of textbooks. Kluwer Academic Publishers. Van den Heuvel-Panhuizen, M. (2005). The role of context in assessment problems in mathematics. For the Learning of Mathematics, 25(2), 2-23. Verschaffel, L., Greer, B., & De Corte, E. (2000). Making sense of word problems. Swets & Zeitlinger. Wijaya, A., van den Heuvel-Panhuizen, M., & Doorman, M. (2015). Opportunity-to-learn context-based tasks provided by mathematics textbooks. Educational Studies in Mathematics, 89(1), 41-65. https://doi.org/10.1007/s10649-015-9595-1 Yam, H. (2005). What is contextual learning and teaching in physics? Retrieved from http://www.phy.cuhk.edu.hk/contextual/approach/tem/brief_e.html