Examining the cut-off score of the English B1 progression exam according to different standard setting methods

Öz

In this study, the cut-off scores obtained from the Angoff, Angoff Y/N, Nedelsky and Ebel standard methods were compared with the 50 T score and the current cut-off score in various aspects. Data were collected from 448 students who took Module B1+ English Exit Exam IV and 14 experts. It was seen that while the Nedelsky method gave the lowest cut-off score, Angoff Y/N method gave the highest cut-off score. The z test was used to determine the difference between the percentages of students who were considered successful according to the methods, and all z values were found to be significant. The classification of students according to their achievement status was examined with the Cohen's Kappa test. Spearman Brown Rank Differences Correlation coefficient was calculated to examine the relationship between the MPSs of the experts according to the methods, and the highest correlation was found between the Angoff-Ebel methods. Wilcoxon test was used to examine the significance of the difference between the MPS of the methods. Because of the test, the difference between Angoff-Nedelsky, Angoff-Ebel, Angoff Y/N-Nedelsky and Nedelsky-Ebel methods was found to be significant. Among the expert decisions, it was seen that there was a moderate level of agreement in the Angoff, and a high level of agreement in the Ebel and Nedelsky methods. A significant difference was found between the current cut-off score, the 50 T score, and the percentages of students considered successful according to the methods.

Anahtar Kelimeler

• Angoff
• Angoff Y/N
• Nedelsky
• Ebel

References

1. Angoff, W.H. (1971). Scales, norms, and equivalent scores. In R.L. Thorndike (Ed.), Educational measurement (2nd ed.). American Council on Education.
2. Berk, R.A. (1986). A consumer’s guide to setting performance standards on criterion-referenced tests. Review of Educational Research, 56(1), 137 172. https://doi.org/10.2307/1170289
3. Boduroğlu, E. (2017). Yükseköğretime geçiş sınavının sınıflama tutarlılığının farklı yöntemlerle elde edilen kesme puanlarına göre incelenmesi [The study of classification consistency of transition to higher education examination according to the cut-off scores obtained from different methods] [Master’s dissertation, Mersin University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
4. Buckhendal, W.C., Smith, W.R., Impara, C.J., & Plake, S.B. (2002). A comprasion of Angoff and Bookmark standard setting methods. Journal of Educational Measurement, 39(3), 253-263. https://doi.org/10.1111/j.1745-3984.2002.tb01177.x
5. Çetin, S. (2011). İşaretleme ve angoff standart belirleme yöntemlerinin karşılaştırılması [Comparison of bookmark and angoff standard setting methods] [Doctoral dissertation, Hacettepe University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
6. Çukadar, İ. (2013). Norm ve ölçüt dayanaklı değerlendirmelerin karşılaştırılmasına ilişkin bir çalışma [A study upon comparison of norm and criterion referenced assessment] [Master's dissertation, Hacettepe University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
7. Demir, O. (2014). Angoff, nedelsky ve ebel standart belirleme yöntemleri ile belirlenen kesme puanlarının karşılaştırılması [A comparison of cutting points determined by angoff, nedelsky and ebel standard setting methods] [Master's dissertation, Abant İzzet Baysal University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
8. Ebel, R.L. (1972). Essentials of educational measurement. Englewood Cliffs NJ: Prentice-Hall.

9. Gündeğer, C. (2012). Angoff, Yes/No ve Ebel Standart Belirleme Yöntemlerinin karşılaştırılması [A comparison of Angoff, Yes/No and Ebel Standard Setting Methods] [Master’s dissertation, Hacettepe University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
10. Hambleton, R.K., Jaeger, R.M., Plake, B.S., & Mills, C. (2000). Setting Performance Standards on Complex Educational Assessments. Applied Psychological Measurement, 24 (4), 355–366. https://doi.org/10.1177/01466210022031804
11. Hambleton, R.K. (2001). Setting performance standards on educational assessments and criteria for evaluating the process. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 89–116). Lawrence Erlbaum Associates Publishers.
12. Hurtz, G.M., & Hertz, N.R. (1999). How many raters should be used for establishing cutoff scores with the Angoff method? a Generalizability Theory Study. Educational and Psychological Measurement, 59 (6), 885 897. https://doi.org/10.1177/00131649921970233
13. Impara, J.C., & Plake, B.S. (1997). Standard setting: An alternative approach. Journal of Educational Measurement, 34(4), 353 366. https://doi.org/10.1111/j.1745 3984.1997.tb00523.x
14. İlhan, M. (2022). Korelasyon [Correlation]. In Çetin B. (Ed.), Eğitimde ölçme ve değerlendirme [Measurement and evaluation in education]. (2nd ed., pp. 23-43). Anı Publishing.
15. Jaeger, R.M. (1989). Certification of student competence. In R.L. Linn (Ed.), Educational measurement (3rd ed. pp 485-514). Macmillan Publishing Co, Inc; American Council on Education.
16. Kılıç, A. (2018). Angoff, yes/no ve sınır grup yöntemlerine göre kesme puanlarının karşılaştırılması [Comparison of cutting points by Angoff, yes / no and borderline group methods] [Master's dissertation, Abant İzzet Baysal University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
17. Kaya, R. (2023). İngilizce B1 seviye atlama sınavının kesme puanının farklı standart belirleme yöntemlerine göre incelenmesi [Examination of the cutting score of the English B1 leveling exam according to different standard determination methods] [Master's dissertation, Gaziantep University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
18. Landis, J.R., & Koch, G.G. (1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33(2), 363-374. https://doi.org/10.2307/2529786
19. Livingston, S.A., & Zieky, M.J. (1983). A comparative study of standard-setting methods. ETS Research Report Series, 1983(2), i 48. https://doi.org/10.1002/j.2330 8516.1983.tb00038.x
20. Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, 14(1), 3-19. https://doi.org/10.1177/001316445401400101
21. Norcini, J.J. (2003). Setting standards on educational tests. Medical education, 37(5), 464-469. https://doi.org/10.1046/j.1365-2923.2003.01495.x
22. Norcini, J.J., Lipner, R.S., Langdon, L.O., & Strecker, C.A. (1987). A Comparison of Three Variations on a Standard‐Setting Method. Journal of Educational Measurement, 24(1), 56–64. https://doi.org/10.1111/j.1745-3984.1987.tb00261.x
23. Ömür, S., & Selvi, H. (2010). Angoff, Ebel ve Nedelsky yöntemleriyle belirlenen kesme puanlarının sınıflama tutarlılıklarının karşılaştırılması. Journal of Measurement and Evaluation in Education and Psychology, 1(2), 109 113. https://dergipark.org.tr/tr/download/article-file/65991
24. Rovai, A., Baker, J., & Ponton, M. (2014). Social science research design and statistics: a practitioner’s guide to research methods and ibm spss analysis. Chesapeake, VA: Watertree Press LLC.
25. Şahin, T. (2019). Nedelsky, sınır grup ve karşıt gruplar standart belirleme yöntemlerinin norma dayalı değerlendirmelerle karşılaştırılması [Comparison of nedelsky, borderline group and constrasting groups standard setting models with norm referenced assessment] [Master's dissertation, Hacettepe University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
26. Tanrıverdi, S. (2006). Standart belirleme yöntemlerinin geçme puanları üzerine etkisi [Impacts of standard setting methods over passing] [Master's dissertation, Hacettepe University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
27. Taşdelen, G. (2009). Nedelsky ve Angoff standart belirleme yöntemlerinin genellenebilirlik kuramı ile karşılaştırılmasına ilişkin bir araştırma [A comparison of Angoff and Nedelsky cutting score procedures using generalizability theory] [Master's dissertation, Hacettepe University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
28. Taşdemir, F. (2013). Angoff (1-0), Nedelsky ve sınır değerleri saptama yöntemleri ile bir testin sınıflama doğruluklarının incelenmesi [Angoff (1-0), Nedelsky and examination of classification accuracies of a test by determination methods of limit values] [Doctoral dissertation, Ankara University]. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
29. Wang, L., Pan, W., & Austin, J.T. (2003). Standards – setting procedures in accountability research: Impacts of conceptual frameworks and mapping procedures on passing rates. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
30. Yıldırım Kan, N. (2019). İngilizce hazırlık atlama sınavı için kesme puanı belirlenmesinde standart belirleme yöntemlerinin karşılaştırılması [Comparing standard setting methods while determining cut point for English proficiency exam] [Master's dissertation, Hacettepe University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
31. Zieky, M.J., & Livingston, S.A. (1977). Basic skills assessment: Manual for setting standards on the Basic Skills Assessment tests. Educational Testing Service.