Comparison of International TIMSS 2011 Proficiency Levels and Cut-off Scores Set by Using Cluster Analysis

Year 2019, , 155 - 168, 17.06.2019

Mahmut Sami Koyuncu , Ayşenur Erdemir

https://doi.org/10.33200/ijcer.544194

Abstract

The aim of the study is to compare TIMSS 2011 proficiency levels with the proficiency levels defined by the researchers using cluster analysis for Turkey, Korean, Norway, and Morocco in 4th and 8th grades in the fields of science and mathematics. Therefore, it is tried to be reached that these cut-off scores for each country can serve the evaluation of each country itself. For this research, the data gathered from related countries’ students was taken from TIMSS 2011 database. Statistical analysis was performed with SPSS Version 21.0 statistic software package. The cut-off scores for these four countries selected in this study for each grade level and course type were defined using cluster analysis. Then, proficiency levels according to these cut-off scores were compared to the general TIMSS 2011 proficiency levels, and so the difference between these levels and percentage of agreement have been examined. According to the results, cut-off scores set by using cluster analysis for Korea were higher than TIMSS international benchmarks. Cut-off scores set for Morocco, Norway, and Turkey were lower than TIMSS international benchmarks. the percentage of agreement of the proficiency levels was found to be between 8.1% and 70.0%, and in general, it has been found that the percentage of agreement was low. Consequently, it is suggested that countries should make a standard-setting study for their own samples instead of using TIMSS international benchmarks for their own evaluations.

Keywords

Standard Setting, Cluster Analysis, Validity, TIMSS2011

References

• Berberoğlu, G. (2009). Madde Haritalama Yöntemi ve Cito Türkiye Öğrenci İzleme Sistemi (ÖİS) Uygulamalarında Yeterlik Düzeylerinin Belirlenmesi. Cito Eğitim: Kuram ve Uygulama Dergisi, Mayıs-Haziran 2009, 13-24.
• Cizek, G. (2001). Conjectures on the rise and call of standard-setting: An introduction to context and practice. In G. J. Cizek (Ed.), Setting performance standards: Concepts,methods, and perspectives (pp. 3–17). Mahwah, NJ: Erlbaum.
• Crocker, L. ve Algina, J. (1986). Intruduction to Classical and Modern Test Theory. New York: CBS.
• Dinçer, E. (2006). Veri Madenciliğinde K-MEANS Algoritması ve Tıp Alanında Uygulanması. Kocaeli Üniversitesi, Fen Bilimleri Enstitüsü, Yüksek Lisans Tezi.
• Gündeğer, C., Doğan, N. (2014). Standart Belirlemede İstatistiksel Bir Yaklaşım: Kümeleme Analizi, IV. Ulusal Eğitimde Ve Psikolojide Ölçme ve Değerlendirme Kongresi (Sözlü Bildiri), 9-13 Haziran 2014, Ankara Hacettepe.
• Hess, B., Subhiyah, R. G., & Giordano, C. (2007). “Convergence Between Cluster Analysis and the Angoff Method for Setting Minimum Passing Scores on Credentialing Examinations”. Evaluation and the Health Professions, 30 (4), 362-375.
• Mullis, I. V., Martin, M. O., Foy, P., & Arora, A. (2012). TIMSS 2011 international results in mathematics. International Association for the Evaluation of Educational Achievement. Herengracht 487, Amsterdam, 1017 BT, The Netherlands.
• Sireci, S. G., Robin, F., & Patelis, T. (1999). Using cluster analysis to facilitate standard setting. Applied Measurement in Education, 12(3), 301-325.
• Violato, C., Marini, A., & Lee, C. (2003). A validity study of expert judgment procedures for setting cutoff scores on high-stakes credentialing examinations using cluster analysis. Evaluation & the health professions, 26(1), 59-72.