A Statistical Approach to Improve the Standard Setting Process: Cluster Analysis

Year 2018, Volume: 8 Issue: 2, 387 - 397, 17.09.2019

Ceylan Gündeğer , Nuri Doğan

Abstract

This study aims to use k-means cluster analysis to improve the standard setting process, to determine the passing score in two samples by three methods, to examine the validity of the cluster analysis results using an external criteria, and to compare the cluster analysis results with those produced by Angoff, Yes/No, and Ebel test centered methods. In this study, the main sample consisted of 305 students and the validation sample consisted of 179 students. The data set is composed of the students' responses to a 20-item achievement test. On the other hand, the number of judges determining the passing score according to the test centered standard setting methods was 17. A moderate correlation was found between the external criteria and the results of cluster analysis for the validation sample. Medium and highly significant relationships were observed between the different statistical methods for determining the passing score. According to the study results, in order to achieve the highest relationship with the test centered standard setting methods’ results, the following methods could be proposed respectively: determine the passing score based on the range comprising the lowest score of the first cluster and the highest score of the second cluster; logistic regression and the average score of successful cluster.

Keywords

Cluster analysis, k-means, passing score

References

• Alpar, R. (2011). Uygulamalı Çok Değişkenli İstatistiksel Yöntemler. Ankara: Detay Yayıncılık Downing, S. M., Tekian, A. & Yudkowsky, R. (2006). Research Methodology: Procedures forEstablishing Defensible Absolute Passing Scores on Performance Examinations in Health Professions Education. Teaching and Learning in Medicine, 18 (1), 50–57. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/16354141 Ebel, R. L. (1972). Essentials of Educational Measurement. New Jersey: Prentice Hall. Gündeğer, C. (2012). Angoff, Yes/No ve Ebel Standart Belirleme Yöntemlerinin Karşılaştırılması (Master’s Thesis, Hacettepe Üniversitesi, Ankara, Turkey) Retrieved from https://tez.yok.gov.tr/UlusalTezMerkezi/ Hess, B., Subhiyah, R. G. & Giordano, C. (2007). Convergence between cluster analysis and the Angoff method for setting minimum passing scores on credentialing examinations. Evaluation and the Health Professions, 30, 362-375. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/17986670 Jaeger, R. M. (1989). Certification of student competence. In R. L. Linn (Ed.), Educational measurement (pp. 485–514). Washington, DC: American Council on Education. Khalid, M. N. (2011). Cluster analysis–a standard setting technique in measurement and testing. Journal of Applied Quantitative Methods, 6, 46–58. Retrieved from http://www.jaqm.ro/issues/volume-6,issue 2/pdfs/6_khalid_shafiq_khan.pdf Livingston, S. A. & Zieky, M. J. (1982). Passing Scores (A Manual for Setting Standards of Performance on Educational and Occupational Tests). New Jersey: Educational Testing Service. Sireci, S. G. (2001). Standard setting using cluster analysis. In G. J. Cizek (Ed.), Setting Performance Standards: Concepts, Methods and Perspectives (pp. 339-354). Mahwah, NJ: Lawrence Erlbaum Associates. Sireci, S. G., Robin, F. & Patelis, T. (1999). Using cluster analysis to facilitate standard setting. Applied Measurement in Education, 12, 301-325. Retrieved from http://www.tandfonline.com/doi/abs/10.1207/S15324818AME1203_5?journalCode=hame20#.VxX4EvmLTIU Şencan, H. (2005). Sosyal ve Davranışsal Ölçümlerde Güvenirlik ve Geçerlik. Ankara: Seçkin Yayıncılık Violato, C., Marini, A., & Lee, C. (2003). A validity study of expert judgment procedures for setting cutoff scores on high-stakes credentialing examinations using cluster analysis. Evaluation and the Health Professions, 26, 59-72. Retrieved from http://www.internationalgme.org/Resources/Pubs/Validity%20Cutoff%20Scores%20-%20Violato.pdf Zieky, M. J. (2001). So Much Has Changed: How the Setting of Cut-scores Has Evolved Since the 1980s. In G. J. Cizek (Ed.), Setting Performance Standards: Concepts, Methods and Perspectives (pp. 19–51). Mahwah, NJ: Lawrence Erlbaum Association