Scoring open-ended items using the fuzzy topsis method and comparing it with traditional approaches

Abstract

This study investigates the application of the fuzzy logic method for scoring open-ended items, specifically comparing its effectiveness against traditional scoring methods. Utilizing the fuzzy TOPSIS method within the mathematics domain, this research established seven criteria for evaluating open-ended responses, developed in consultation with three experts. Due to constraints imposed by the pandemic, the study did not proceed with a real-world application; instead, it simulated data for 25 students to compare the rankings derived from traditional and fuzzy logic methods using the MS Excel program. The research produced three distinct rankings using the conventional method and analyzed the correlation between these rankings and those generated by the fuzzy TOPSIS method, employing the Spearman rank correlation coefficient. The findings reveal a significantly positive correlation between the rankings obtained through traditional methods and those acquired via the fuzzy logic approach, suggesting the latter's potential as an effective alternative for evaluating open-ended responses.

Keywords

• Fuzzy logic
• Open-ended items
• Topsis
• Fuzzy topsis

References

1. Altun, M. (2002). İlköğretim ikinci kademede matematik öğretimi [Mathematics teaching in the second level of primary education]. Alfa Yayınevi.
2. Armağan, H. (2008). A new approach for student academic performance evaluation [Unpublished Master’s thesis]. Süleyman Demirel University.
3. Arslan, M. (2019). Evaluation of teacher performances with fuzzy logic method [Unpublished Master’s thesis]. Van Yüzüncü Yıl University.
4. Avcı Öztürk, B. (2018). Analitik hiyerarşi süreci ve topsis: Bulanık uygulamaları ile [Analytic hierarchy process and topsis: With fuzzy applications]. Dora Basın Yayın Dağıtım.
5. Badger, E., & Thomas, B. (1992). Open-ended questions in reading. Practical Assessment, Research & Evaluation, 3(4), 03. https://doi.org/10.7275/fryf-z044
6. Bakanay, D. (2009). The assessment of micro-teaching performance by fuzzy logic. [Unpublished Master’s thesis]. Marmara University.
7. Baykal, N., Beyan, T. (2004). Bulanık mantık ilke ve temelleri [Principles and fundamentals of fuzzy logic]. Bıçaklar Kitabevi.
8. Bostan, A. (2017). Measurement in criterion based tests with utilization of fuzzy logic and usage of this methodology in computerized adaptive tests [Unpublished Doctoral thesis]. Gazi University.

9. Bush, M. (2001). A multiple choice test that rewards partial knowledge. Journal of Further and Higher Education, 25(2), 157–163. https://doi.org/10.1080/03098770120050828
10. Chen, T.C. (2000). Extensions of the topsis for group decision - making under fuzzy environment. Fuzzy Sets And Systems, 114, 1-9. https://doi.org/10.1016/S0165-0114(97)00377-1
11. Cheng, C.H. (1996). Evaluating naval tactical missile systems by fuzzy ahp based on the grade value of membership function. Europan Journal of Operational Research, 96(2), 343-350. https://doi.org/10.1016/S0377-2217(96)00026-4
12. Cooney, T.J., Sanchez, W.B., Leatham, K., & Mewborn, D.S. (2004). Open-ended assessment in math: A searchable collection of 450+ questions.
13. Çakar, T. (2020). Bulanık çok ölçütlü karar verme yöntemleri [Multicriteria fuzzy decision making methods]. İstanbul Gelişim Üniversitesi Yayınları.
14. Coşkunırmak, Y. (2010). Fuzzy linear programming and an application of fuzzy goal programming in local governments [Unpublished Master’s thesis]. Çukurova University.
15. Damlar Demirci, P.(2019). The investigation of open-ended items scoring methods by generalizability theory [Unpublished Master’s thesis]. Ege University.
16. Dündar, S., Fatih, E., & Özdemir, Ş. (2007). Fuzzy topsis yöntemi ile sanal mağazalarin web sitelerinin değerlendirilmesi [Evaluation of web sites of virtual stores with fuzzy topsis method]. Atatürk Üniversitesi İktisadi ve İdari Bilimler Dergisi, 21(1), 287-305. https://dergipark.org.tr/en/pub/atauniiibd/issue/2691/35409
17. Ecer, F. (2007). Assessing candidates in human resource selection with the Fuzzy Topsis method and an application [Unpublished Doctoral thesis]. Afyon Kocatepe University.
18. Elmas, Ç. (2003). Bulanık mantık denetleyiciler: Kuram, uygulama, sinirsel bulanık mantık [Fuzzy logic controllers: Theory, application, neural fuzzy logic]. Seçkin Yayıncılık.
19. Elmas, Ç. (2011). Yapay zeka uygulamaları [Artificial intlligence applications]. Seçkin Yayıncılık.
20. Erdin, C. (2007). Fuzzy goal programming and an applied study in management [Unpublished Doctoral thesis]. İstanbul University.
21. Eroğlu, G. (2012). The transition from classical logic to modern logic: some of the foundations grounding the rise of modern logic.Hikmet Yurdu Düşünce - Yorum Sosyal Bilimler Araştırma Dergisi, 5(9), 115-135. http://dx.doi.org/10.17540/hy.v5i9.167
22. Ertuğrul, İ. (2006). Akademik performans değerlendirmede bulanık mantık yaklaşımı [Fuzzy logic approach in academic performance evaluation]. İktisadi ve İdari Bilimler Dergisi, 20(1), 155-176. https://dergipark.org.tr/en/pub/atauniiibd/issue/2689/35353
23. Ertuğrul, İ., & Karakaşoğlu, N. (2008). Banka şube performasnlarının VIKOR yöntemi ile değerlendirilmesi [Evaluation of bank branch performances with VIKOR method]. Endüstri Mühendisliği dergisi, 20(1), 19 28. https://www.mmo.org.tr/sites/default/files/c4692732b25c1ee_ek.pdf
24. Geer, J.G. (1988). What do open-ended questions measure? Public Opinion Quarterly, 52(3), 365–367. https://doi.org/10.1086/269113
25. Güler, O., & Yücedağ, İ. (2017). Fuzzy logic-based approach to site selection problem of vocational secondary school students. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi, 32(1), 111-122. https://doi.org/10.16986/HUJE.2016018727
26. Haladyna, T.M. (1997). Writing tests items to evaluate higher-order thinking. Allyn & Bacon.
27. Hasırcı, N. (2010). İbn Teymiyye’nin mantık eleştirisi [İbn Teymiyye’s criticism of logic]. Araştırma Yayınları.
28. Hocalar, E. (2007). An application of fuzzy balanced scorecard system for managing performance in higher education organizations [Unpublished Master’s thesis]. Sakarya University.
29. Hwang, C.L., & Yoon, P. (1981). Multiple attribute decision making in: lecture notes in economics and mathematical systems. Springer-Verlag, Berlin.
30. Kaptanoğlu, D., & Özok, A.F. (2006). Akademik performans değerlendirmesi için bir bulanık model [A fuzzy model for academic performance evaluation]. İTÜ dergisi, 5 (1), 193-204. http://itudergi.itu.edu.tr/index.php/itudergisi_d/article/view/627
31. Karadeniz, A. (2016). Design, evaluation and implementation of a system aimed at assessment of learning achievement through open-ended questions in massive, open and distance learning [Unpublished Doctoral thesis] Anadolu University.
32. Karakaya, İ. (2022). Açık uçlu soruların hazırlanması, uygulanması ve değerlendirilmesi [Preparation, implementation and evaluation of open-ended questions]. Pegem Akademi.
33. Karataş, İ. (2018). Comparison of fuzzy logic, classic logic and symbolic logic. European Journal of Educational & Social Sciences, 3(2), 144 163. https://dergipark.org.tr/en/pub/ejees/issue/40157/477684
34. Köz, İ. (2002). Aristoteles mantığı ile felsefe-bilim ilişkisi [Relationship between philosophy-science and Aristotelian logic]. Ankara Üniversitesi İlahiyat Fakültesi Dergisi, 43(2), 55-37. https://dergipark.org.tr/en/download/article-file/583546
35. Klufa, J. (2018). Multiple choice question tests–advantages and disadvantages. Recent Advances in Educational Technologies. ISBN: 978 1 61804 322 1. https://www.inase.org/library/2015/zakynthos/bypaper/EDU/EDU-07.pdf
36. Kutlu, Ö., Doğan, C.D., & Karakaya, İ. (2014). Öğrenci başarısının değerlendirilmesi: Performansa ve portfolyoya dayalı durum belirleme [Assessment of student achievement: Performance and portfolio-based assessment]. Pegem Akademi.
37. Madi, E.N., Garibaldi, J.M., & Wagner, C. (2017, July). Exploring the use of type-2 fuzzy sets in multi-criteria decision-making based on TOPSIS. In 2017 IEEE International Conference on Fuzzy Systems (FUZZ IEEE) (pp. 1 6). IEEE. https://doi.org/10.1109/FUZZ-IEEE.2017.8015664
38. McMillan, J.H. (2017). Classroom assessment: Principles and practice that enhance student learning and motivation. Pearson.
39. McNeill, F.M., & Ellen T. (1994). Fuzzy Logic: A practical approach. Academic Press.
40. MEB. (2017). 8. sınıf merkezi ortak sınavları matematik dersi açık uçlu soru ve yapılandırlmış cevap anahtarı örnekleri [8th grade central common exams mathematics open-ended questions and structured answer key examples]. MEB http://odsgm.meb.gov.tr/meb_iys_dosyalar/2017_09/15135732_Mat_acik_uclu.pdf
41. MEB. (2020). Milli eğitim istatistikleri: Örgün eğitim 2019-20 [National education statistics: Formal education 2019-20]. Türkiye İstatistik Kurumu.
42. MEB. (2020). Ortaöğretim kurumlarına ilişkin merkezi sınav kılavuzu [Central exam guide for secondary education]. Eğitim, Analiz ve Değerlendirme Raporları Serisi.
43. Miller, M.D., Linn, R., & Gronlund, N.E. (2009). Measurement and assessment in teaching. Merrill Prentice Hall.
44. Nitko, A.J., & Brookhart, S.M. (2014). Education assessment of students. Merrill Prentice Hall.
45. Opricovic, S., &Tzeng, G. H. (2004). Compromise solution by mcdm methods: A comparative analysis of vıkor and topsıs. European Journal of Operational Research, 156(2), 445-455. https://doi.org/10.1016/S0377-2217(03)00020-1
46. Öksüz, Y., & Güven Demir, E. (2019). Comparison of open ended questions and multiple choice tests in terms of psychometric features and student performance. Hacettepe University Journal of Education, 34(1), 259 282. https://doi.org/10.16986/HUJE.2018040550
47. Öner, N. (1986). Klasik mantık [Classical logic]. Ankara Üniversitesi Basımevi.
48. ÖSYM. (2017). Açık uçlu sorular hakkında bilgilendirme ve açık uçlu soru örnekleri [Information about open-ended questions and examples of open-ended questions]. OSYM https://www.osym.gov.tr/TR,12909/2017-lisans-yerlestirme-sinavlari-2017-lys-acik-uclu-sorular-hakkinda-bilgilendirme-ve-acik-uclu-soru-ornekleri-05012017.html
49. ÖSYM. (2021). 2021 yılı yükseköğretim kurumları sınavı (YKS) kılavuzu [2021 higher education institutions exam guide]. OSYM.
50. Özdağoğlu, A. (2016). Bulanık işlemler, durulaştırma ve sözel eşikler: Bilgisayar uygulamalı örneklerle [Fuzzy operations, stabilization and verbal thresholds: With computer-implemented examples]. Detay Yayıncılık.
51. Paksoy, T., Pehlivan, N.Y., & Özceylan, E. (2013). Bulanık küme teorisi [Fuzzy set theory]. Nobel Akademik Yayıncılık.
52. Popham, W. J. (1999). Classroom assessment: What teachers need to know. Allyn & Bacon.
53. Royal, K.D., & Hecker, K.G. (2016). Rater errors in clinical performance assessments. Journal of Veterinary Medical Education, 43(1), 5-8. https://doi.org/10.3138/jvme.0715-112R
54. Ross, T.J. (2010). Fuzzy logic with engineering applications. John Wiley & Sons Ltd.
55. Sarı, M., Murat, Y.S., & Kırabalı, M. (2005).Fuzzy modeling approach and applications. Dumlupınar Üniversitesi Fen Bilimleri Enstitüsü Dergisi 9, 77 92. https://dergipark.org.tr/en/pub/pufbed/issue/36210/407845
56. Taylan, N. (2008). Ana hatlarıyla mantık [Logic in outline]. Ensar Neşriyat Yayıncılık.
57. Tekin, H. (2010). Eğitimde ölçme ve değerlendirme [Measurement and evaluation in education]. Yargı Yayınevi.
58. Topçu, H. (2014). Examination of fuzzy AHP method and an application on the problem of reference book selection to KPSS preparations [Unpublished Master’s thesis]. Marmara University.
59. Turgut, M.F., & Baykul, Y. (2012). Eğitimde ölçme ve değerlendirme [Measurement and evaluation in education]. Pegem Akademi.
60. Türe, H. (2006). Fuzzy linear programming and an application [Unpublished Master’s thesis]. Gazi University.
61. Tzeng, G.H., & Huang, J.J. (2011). Multiple attribute decision making: methods and applications. CRC press.
62. Uygunoğlu, T., & Osman, Ü. (2005). Fuzzy logic approach on the effect of seyitömer fly ash on compressive strength of concrete. Yapı Teknolojileri Elektronik Dergisi, 1(1), 13-20. https://dergipark.org.tr/en/pub/yted/issue/22222/238558
63. Van de Walle, J.A., Karp, K.S., & Bay-Williams, J.M. (2014). Elementary and middle school mathematics. Pearson.
64. Wimatsari, G.A. Ketut, G.P., & Buana, P.W. (2013). Multi-attribute decision-making scholarship selection using a modified fuzzy topsis. International Journal of Computer Science Issues, 10(1), 309 317. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=7f6d03cd62a7d6c4f80eca3b27c788c6a5000a5d
65. Yazırdağ, M. (2018). Supply system with fuzzy AHP and fuzzy TOPSIS methods: An application in gendarmerie [Unpublished Master’s thesis]. Gazi University.
66. Yen, J., & Langari, R. (1999). Fuzzy logic: intelligence, control, and information. Prentice Hall.
67. Yılmaz, R. (2008). Student selection for postgraduate education in Turkey: an empirical study at Turkish Military Academy Defense Sciences Institute [Unpublished Master’s thesis]. Kara Harp Okulu.
68. Zadeh, L.A. (1965). Fuzzy sets. Information and Control, 8, 338 353. https://doi.org/10.1016/S0019-9958(65)90241-X
69. Zimmermann, H. J. (1978). Fuuzy programming and linear programming with several objetive functions. Fuzzy Sets and Systems, 1(1), 45-55 https://doi.org/10.1016/0165-0114(78)90031-3
70. Zimmermann, H.J. (2001). Fuzzy set theory-and its applications. Springer Science.