ROC, Doğrudan Uzlaşma ve Angoff Yöntemleriyle Belirlenen Kesme Puanlarının Karşılaştırılması

Abstract

Araştırmada Reciever Operating Characteristic (ROC), Angoff ve Doğrudan Uzlaşma Yöntemi (DUY) standart belirleme yöntemlerinden elde edilen kesme puanları karşılaştırılmıştır. Bu amaç doğrultusunda “Ege Üniversitesi Öğrenci Memnuniyeti Ölçeği” için ROC, Angoff ve DUY ile kesme puanı belirlenmiştir. Araştırma örneklemi 2021’de kayıtlı 23655 Ege Üniversitesi öğrencilerinden ve Üniversitesi Psikoloji bölümünde görev yapan 10 uzmandan oluşmaktadır. Araştırmadaki uzman görüşleri doğrultusunda Angoff için belirlenen kesme puanı 51,60; DUY için 49,50 olarak bulunmuştur. Uzman görüşü alınmayan ve altın standardı bulunmayan ROC için Örtük Sınıf Analizi yöntemiyle oluşturulan sınıflar altın standart olarak kullanıldığındaki kesme puanı 57 olarak bulunmuştur. Kesme puanlarına göre öğrencileri memnun/memnun değil sınıflandırmasında Cohen Kappa katsayısı ve z testi hesaplanmıştır. Cohen Kappa katsayıları ROC-Angoff için 0,66 ile iyi; DUY-Angoff için 0,91 ile mükemmel; DUY-ROC için 0,59 ile orta uyum bulunmuştur. Z testi sonuçlarında ise ROC-Angoff, ROC-DUY, DUY-Angoff arasında anlamlı farklılık bulunmuştur. Uzman kesme puanlarının uyumu için hesaplanan Kendall W katsayısı Angoff (0,14), DUY (0,27) için uzmanların uyumsuz olduğu bulunmuştur.

Keywords

• Standart belirleme
• ROC
• Angoff
• doğrudan uzlaşma
• kesme puanı
• altın standartsız ROC yöntemi

Ethical Statement

Araştırma, Ege Üniversitesi Sosyal ve Beşeri Bilimler Bilimsel Araştırma ve Yayın Etik Kurulu tarafından 07/10 sayılı toplantı/karar ile etik kurul onayı oybirliğiyle alınarak yürütülmüştür (Protokol No: 1557/ Tarih: 27/07/2022). Araştırmada yer alan tüm katılımcılardan ve uzmanlardan gönüllü ve bilgilendirilmiş onam alınmıştır.

Comparison of Cutoff Scores Determined by ROC, Direct Consensus and Angoff Methods

Abstract

The study compared cut-off scores obtained from the Receiver Operating Characteristic (ROC), Angoff Method, and Direct Consensus Method (DCM). Cut-off scores for the “Ege University Student Satisfaction Scale” were determined using these methods. The sample consisted of 23,655 students enrolled at Ege University in 2021 and 10 experts from the Department of Psychology. Based on expert judgments, the cut-off score was 51.60 for Angoff and 49.50 for DCM. In the ROC analysis, where no expert judgment and external gold standard was available, classes derived from Latent Class Analysis were used as the gold standard, resulting in a cut-off score of 57. Student classifications(satisfied/not satisfied) were compared using the Cohen’s Kappa coefficient and the z test. Kappa values indicated good agreement for ROC–Angoff (0.66), excellent agreement for DCM–Angoff (0.91), and moderate agreement for DCM–ROC (0.59). Z-test results showed significant differences among the methods. Kendall’s W indicated low expert agreement for Angoff (0.14) and DCM (0.27).

Keywords

• Standard setting
• reciever operating characteristic
• ROC
• Angoff
• direct consensus
• cut-off score
• ROC without gold standard

Ethical Statement

Araştırma, Ege Üniversitesi Sosyal ve Beşeri Bilimler Bilimsel Araştırma ve Yayın Etik Kurulu tarafından 07/10 sayılı toplantı/karar ile etik kurul onayı oybirliğiyle alınarak yürütülmüştür (Protokol No: 1557/ Tarih: 27/07/2022). Araştırmada yer alan tüm katılımcılardan ve uzmanlardan gönüllü ve bilgilendirilmiş onam alınmıştır.

References

1. Abd-Rahman, A. N. A., Baharuddin, I. H., Abu-Hassan, M. I., & Davies, S. J. (2021). A comparison of different standard-setting methods for professional qualifying dental examination. Journal of Dental Education, 85(7), 1210–1216. https://doi.org/10.1002/jdd.12600
2. Afrashteh, M. (2021). Comparison of the validity of bookmark and Angoff standard setting methods in medical performance tests. BMC Medical Education, 21(1). https://doi.org/10.1186/s12909-020-02436-3
3. Akbari, A., & Shahrokhi, M. (2024). Unveiling fairness in scoring: A thorough method for precise cutoff score calculation in education assessment. Quality Assurance in Education, 32(3). https://doi.org/10.1108/qae-12-2023-0208
4. Alkhaldi, H., Alkhutaba, M., Almassarweh, S., Al-Mistareehi, H. H., & Al Yamani, A. R. H. (2023). Building a criterion referenced test in measurement and evaluation and determining its cut off score by several methods. Journal of Education and Practice, 14(1), 42–61. https://doi.org/10.7176/JEP/14-1-06
5. Alonzo, T. A., & Pepe, M. S. (1999). Using a combination of reference tests to assess the accuracy of a new diagnostic test. Statistics in Medicine, 18, 2987–3003. https://doi.org/10.1002/sim.762
6. Alpar, R. (2010). Spor, sağlık ve eğitim bilimlerinden örneklerle uygulamalı istatistik ve geçerlik- güvenirlik. Detay Yayıncılık.
7. Angoff, W. H. (1971). Scales, norms and equivalent scores. R.L. Thorndike (Haz.), Educational Measurement (2. baskı). American Council on Education.
8. Bahadır, E., & Kalender, B. (2018). Roc analizi ile Zung depresyon ölçeği kesme noktalarının belirlenmesi. Adnan Menderes Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, 5(2), 137–147. https://doi.org/10.30803/adusobed.407400

9. Baik, Y., Rickman, H., Hanrahan, C., Mmolawa, L., Kitonsa, P., Sewelana, T., Nalutaaya, A., Kendall, E., Lebina, L., Martinson, N., Katamba, A., & Dowdy, D. (2020). A clinical score for identifying active tuberculosis while awaiting microbiological results: Development and validation of a multivariable prediction model in sub-Saharan Africa. PLoS Medicine, 17. https://doi.org/10.1371/journal.pmed.1003420
10. Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246. https://doi.org/10.1037/0033-2909.107.2.238
11. Bogowicz, M., Jochems, A., Deist, T. M., Tanadini-Lang, S., Huang, S. H., Chan, B., Waldron, J. N., Bratman, S., O'Sullivan, B., Riesterer, O., Studer, G., Unkelbach, J., Barakat, S., Brakenhoff, R. H., Nauta, I., Gazzani, S. E., Calareso, G., Scheckenbach, K., Hoebers, F., Wesseling, F. W. R., … Lambin, P. (2020). Privacy-preserving distributed learning of radiomics to predict overall survival and HPV status in head and neck cancer. Scientific Reports, 10(1), 4542. https://doi.org/10.1038/s41598-020-61297-4
12. Brennan, R., & Lockwood, R. (1980). A Comparison of the nedelsky and angoff cutting score procedures using generalizability theory. Applied Psychological Measurement, 4, 219–240. https://doi.org/10.1177/014662168000400209
13. Buckhendal, W. C., Smith. W. R., Impara, C. J., & Plake. S. B. (2002). A comprasion of Angoff and Bookmark standard setting methods. Journal of Educational Measurement, 39(3), 253–263. https://doi.org/10.1111/j.1745-3984.2002.tb01177.x
14. Chang, L. A. (1996). Comparison between the Nedelsky and Angoff standart setting methods. The Annual Meeting of the National Council on Measurement in Education. 1-32.
15. Cheung, A., Dufour, S., Jones, G., Kostoulas, P., Stevenson, M. A., Singanallur, N. B., & Firestone, S. M. (2021). Bayesian latent class analysis when the reference test is imperfect. Revue Scientifique et Technique (International Office of Epizootics), 40(1), 271–286. https://doi.org/10.20506/rst.40.1.3224
16. Cizek, G. J., & Bunch, M. B. (2007). Standart Setting: A guide to establishing and evaluating performance standards on tests. Sage Publications.
17. Clauser, B., Kane, M., & Clauser, J. (2020). Examining the precision of cut scores within a generalizability theory framework: A closer look at the item effect. Journal of Educational Measurement, 57(2), 216–229. https://doi.org/10.1111/jedm.12247
18. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational And Psychological Measurement, 20, 37–46. https://doi.org/10.1177/001316446002000104
19. Cronbach, L. J. (1990). Essentials Of Psychological Testing (5.baskı). Harper & Row.
20. Çetin, S., & Gelbal, S. (2010). Impact of standard setting methodologies over passing scores. Ankara University Journal of Faculty of Educational Sciences, 43(1), 79–96. https://doi.org/10.1501/Egifak_0000001191 dergipark.org.tr
21. Demir, O., & Köse, İ. (2014). Angoff, Nedelsky ve Ebel standart belirleme yöntemleri ile belirlenen kesme puanlarının karşılaştırılması. Mersin Üniversitesi Eğitim Fakültesi Dergisi, 10(2), 14–27.
22. Dirican, A. (2001). Tanı testi performansının değerlendirilmesi ve kıyaslanması. Cerrahpaşa Tıp Dergisi, 32(1), 25–30.
23. Dorfman, D. D., & Alf, E. (1968). Maximum likelihood estimation of parameters of signal detection theory-a direct solution. Psychometrika, 33(1), 117–124. https://doi.org/10.1007/BF02289677
24. Eckes, T. (2017). Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis. Language Testing, 34, 383–411. https://doi.org/10.1177/0265532216672703
25. Eleuteri, A., Rola, A., Kalirai, H., Hussain, R., Sacco, J., Damato, B., Heimann, H., Coupland, S., & Taktak, A. (2021). Cost-utility analysis of a decade of liver screening for metastases using the Liverpool Uveal Melanoma Prognosticator Online (LUMPO). Computers in Biology and Medicine, 130, 104221. https://doi.org/10.1016/j.compbiomed.2021.104221
26. Elmalı, F. (2009). Altın standartlı ve altın standartsız durumlarda parametrik, yarı parametrik ve parametrik olmayan Roc eğrisi yöntemlerinin karşılaştırılması [Doktora Tezi, Eskişehir Osmangazi Üniversitesi]. Yüksek Öğretim Kurulu Tez Merkezi. https://tez.yok.gov.tr/UlusalTezMerkezi/
27. Enøe, C., Georgiadis, M. P., & Johnson, W. O. (2000). Estimation of sensitivity and specificity of diagnostic tests and disease prevalence when the true disease state is unknown. Preventive Veterinary Medicine, 45(1-2), 61–81. https://doi.org/10.1016/s0167-5877(00)00117-3
28. Erkuş, A. (2007). Ölçek geliştirme ve uyarlama çalışmalarında karşılaşılan sorunlar. Türk Psikoloji Bülteni, 13(40), 17.
29. Erkuş, A., & Selvi, H. (2019). Psikolojide ölçme ve ölçek geliştirme III: Ölçek uyarlama ve norm geliştirme. Pegem Akademi.
30. Espeland M. A., & Handelman S.L. (1989). Using latent class models to characterize and assess relative error in discrete measurements. Biometrics, 45, 587–599.
31. Fidelis, I., & Tei-Firstman, R. I. (2017). Comparism of cut-off scores using two setting methods. International Journal of Quantitative and Qualitative Research Methods, 5(2), 42–61.
32. Flach, P. A., Blockeel, H., Ferri, C., Hernández-Orallo, J., & Struyf, J. (2003). Decision support for data mining: Introduction to ROC analysis and its applications. D. Mladenić, N. Lavrač, M. Bohanec, & S. Moyle (Haz.), Data mining and decision support: Integration and collaboration içinde (s. 81–90). Kluwer Academic Publishers. https://doi.org/10.1007/978-1-4615-0286-9_7
33. Grapin, S., Kranzler, J., Waldron, N., Joyce-Beaulieu, D., & Algina, J. (2017). Developing local oral reading fluency cut scores for predicting high-stakes test performance. Psychology in the Schools, 54, 932–946. https://doi.org/10.1002/PITS.22035
34. Green, D. M., & Swets, J.A. (1966). Signal detection theory and psychophysics. Wiley.
35. Gushta, M. M. (2003). Standard-Setting issues in computerized-adaptive testing. The Annual Conference Of The Canadian Society For Studies in Education, 1–36.
36. Gündeğer, C., & Doğan, N. (2014). Angoff, Yes/No ve Ebel standart belirleme yöntemlerinin karşılaştırılması. Journal of Measurement and Evaluation in Education and Psychology, 5(1), 53–60. https://doi.org/10.21031/epod.47091 dergipark.org.trresearchgate.net
37. Hambleton, R. K. (2001). Setting Performance Standards: Concepts, Methods And Perspectives. Lawrence Erlbaum Publishers.
38. Hambleton, R. K., & Plake, B. S. (1995). Using an extended Angoff procedure to set standards on complex performance assessments. Applied Measurement in Education, 8(1), 41–55.
39. Hanley, J. A., & Mcneil, B. J. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148(3), 839–843.
40. Hassanzad, M., & Hajian-Tilaki, K. (2024). Methods of determining optimal cut-point of diagnostic biomarkers with application of clinical data in ROC analysis: An update review. BMC Medical Research Methodology, 24. https://doi.org/10.1186/s12874-024-02198-2
41. Holcomb, M. J. (2012). Psychometric properties of a measure of competency for use in civil legal proceedings (Doktora Tezi, Ball State University). Ball State University Cardinal Scholar.
42. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.
43. Huang, Q., Jiang, Z., Shi, B., Meng, J., Shu, L., Hu, F., & Mi, J. (2025). Characterisation of cardiovascular disease (CVD) incidence and machine learning risk prediction in middle-aged and elderly populations: Data from the China health and retirement longitudinal study (CHARLS). BMC Public Health, 25. https://doi.org/10.1186/s12889-025-21609-7
44. Hui S. L., & Walter S. D. (1980). Estimating the error rates of diagnostic tests. Biometrics, 36(1), 167–171. https://doi.org/10.2307/2530508
45. Hurtz, G. M., & Hertz, N. R. (1999). How many raters should be used for establishing cutoff scores with the Angoff method? A generalizability theory study. Educational And Psychological Measurement, 59(6), 885–897.
46. Impara, J. C., & Plake, B. S. (1997). Standard setting: An alternative approach. Journal Of Educational Measurement, 34(4), 353–366.
47. Jaeger, R. M. (1989). Certification of student competence. R. L. Linn (Haz.) Educational measurement içinde, 485–514. Macmillan.
48. Johnson, W. O., Gastwirth, J. L., & Pearson, L. M. (2001). Screening without a "gold standard": The Hui-Walter paradigm revisited. American Journal Of Epidemiology, 153(9), 921–924. https://doi.org/10.1093/aje/153.9.921
49. Joseph, L., Gyorkos, T. W., & Coupal, L. (1995). Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard. American Journal Of Epidemiology, 141(3), 263–272. https://doi.org/10.1093/oxfordjournals.aje.a117428
50. Karasar, N. (2012). Bilimsel araştırma yöntemi (23. baskı). Nobel Yayıncılık.
51. Kaya, R., & Çeti̇n, B. (2025). Examining the cut-off score of the English B1 progression exam according to different standard setting methods. International Journal of Assessment Tools in Education, 12(1), 78–92. https://doi.org/10.21449/ijate.1371162
52. Kılıç, A. (2018). Angoff, Yes/No ve Sınır Grup yöntemlerine göre kesme puanlarının karşılaştırılması. [Yüksek lisans tezi, Bolu Abant İzzet Baysal Üniversitesi]. Yüksek Öğretim Kurumu Tez İzleme Merkezi. https://tez.yok.gov.tr/UlusalTezMerkezi/
53. Kılıç, M. (2022). Standart Belirleme Yöntemlerine Sınıflama Önerisi. 8. Uluslararası Eğitimde ve Psikolojide Ölçme ve Değerlendirme Kongresi, 1–4. https://kitap.epodder.org/
54. Koyuncu, S. (2015). Psikolojik ölçeklerde Roc analizi yöntemiyle standart belirleme. [Yüksek Lisans Tezi, Gazi Üniversitesi]. Yüksek Öğretim Kurumu Tez İzleme Merkezi. https://tez.yok.gov.tr/UlusalTezMerkezi/
55. Krzanowski, W. J., & Hand, D. J. (2009). Roc curves for continuous data. Chapman And Hall, Crc Press.
56. Landis J. R.., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
57. Lasko, T. A., Bhagwat, J. G., Zou, K. H., & Ohno-Machado, L. (2005). The use of receiver operating characteristic curves in biomedical informatics. Journal Of Biomedical Informatics, 38, 404–415.
58. Linzer, D. A., & Lewis, J . B. (2011). poLCA: An R package for polytomous variable latent class analysis. Journal of Statistical Software, 42(10), 1–29. https://doi.org/10.18637/jss.v042.i10
59. Livingston, S. A., & Zieky, M. J. (1982). Passing Scores: A manual for setting standards of performance on educational and occupational tests. Educational Testing Service.
60. Lusted, L. B. (1971). Signal detectability and medical decision making. Science, 171(3977), 1217–1219.
61. Margolis, M. J., Mee, J., Clauser, B. E., Winward, M., & Clauser, J. C. (2016). Effect of content knowledge on Angoff-Style standard setting judgments. Educational Measurement: Issues And Practice, 35(1), 29–37. https://doi.org/10.1111/emip.12104
62. Marrie, R., Zhang, L., Lix, L., Graff, L., Walker, J., Fisk, J., Patten, S., Hitchon, C., Bolton, J., Sareen, J., El‐Gabalawy, R., Marriott, J., & Bernstein, C. (2018). The validity and reliability of screening measures for depression and anxiety disorders in multiple sclerosis. Multiple Sclerosis And Related Disorders, 20, 9–15. https://doi.org/10.1016/j.msard.2017.12.007
63. McDonald, R. P. (1985). Factor analysis and related methods. Lawrence Erlbaum Associates.
64. MedCalc Software Ltd . (2023). Medcalc (Versiyon 20.113). [Bilgisayar yazılımı] .https://www.medcalc.org/en/index.php
65. Muthén, B. Ö. (1993). Goodness of fit with categorical and other non-normal variables. K. A. Bollen & J. S. Long (Haz.) Testing Structural Equation Models içinde, 205–243. Sage.
66. Muthén, L. K., & Muthén, B. O. (1998–2012). Mplus user’s guide (Version 7) [Computer software]. Muthén & Muthén.
67. Özdamar, K. (2003). SPSS ile biyoistatistik. Kaan Kitabevi.
68. Park, J., Ahn, D. S., Yim, M. K., & Lee, J. (2018). Comparison of standard-setting methods for the Korea radiological technologist licensing examination: Angoff, Ebel, Bookmark, and Hofstee. Journal of Educational Evaluation for Health Professions, 15, 32. https://doi.org/10.3352/jeehp.2018.15.32
69. Pitoniak, M. J. (2002). Investigation of two standard setting methods for a licensure examination [Yüksek Lisans Tezi, University of Massachusetts Amherst]. ScholarWorks@UMass Amherst. https://scholarworks.umass.edu/dissertations/AAI1418240/
70. Pitoniak, M. J. (2003). Standard setting methods for complex licensure examinations [Doktora Tezi, University of Massachusetts Amherst]. ScholarWorks@UMass Amherst. https://scholarworks.umass.edu/dissertations/AAI3110531/
71. Qu, Y., Tan, M., & Kutner, M. H. (1996). Random effects models in latent class analysis for evaluating accuracy of diagnostic tests. Biometrics, 52(3), 797–810. https://doi.org/10.2307/2533043
72. R Core Team (2024). R: A language and environment for statistical computing. R Foundation for statistical computing. https://www.R-project.org/.
73. Reyhanlıoğlu Keçeoğlu, Ç. (2012). Kesme puanının belirlenmesinde lojistik regresyon analizi, diskriminant analizi ve Roc eğrisi yöntemlerinin karşılaştırılması [Yüksek Lisans Tezi, Hacettepe Üniversitesi]. Yüksek Öğretim Kurulu Tez Merkezi. https://tez.yok.gov.tr/UlusalTezMerkezi/
74. Reyhanlıoğlu Keçeoğlu Ç., Gelbal, S., & Doğan, N. (2016). ROC eğrisi yöntemi ile kesme puanının belirlenmesi. The Journal of Academic Social Sciences Studies, 50(2), 553-562. https://doi.org/10.9761/JASSS3564
75. RStudio Team. (2024). RStudio: Integrated Development Environment for R (Version 2024.03.1). RStudio, PBC. https://posit.co/products/open-source/rstudio/
76. Schermelleh Engel, K., Moosbrugger, H., & Muller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness of-fit measures. Methods Of Psychological Research Online, 8(2), 23–74. https://doi.org/10.23668/psycharchives.12784
77. Shapiro, D. E. (1999). The interpretation of diagnostic tests. Statistical Methods İn Medical Research, 8(2),113–134.
78. Sireci, S. G., Hambleton, R. K., Huff, K. L., & Jodoin, M. G. (2000). Setting and validating standards on Microsoft certified professional examinations (Laboratory of Psychometric and Evaluative Research Report No. 395). University of Massachusetts Amherst, School of Education, Laboratory of Psychometric and Evaluative Research.
79. Sireci, S. G., Hambleton, R. K., & Pitoniak, M. J. (2004). Setting passing scores on licensure exams using direct consensus. Clear Exam Review, 15(1), 21–25.
80. Sireci, S. G., Pitoniak, M. J., Meara, K. C., & Hambleton, R. K. (2000). A review of standard setting methods applicable to the Advanced Placement Examination Program (Laboratory of Psychometric and Evaluative Research Report No. 375). University of Massachusetts Amherst, School of Education, Laboratory of Psychometric and Evaluative Research.
81. Son, D., Lee, H. S., Lee, Y., Lee, J., & Han, J. (2021). Comparison of triglyceride-glucose index and HOMA-IR for predicting prevalence and incidence of metabolic syndrome. Nutrition, Metabolism, and Cardiovascular diseases: NMCD, 32(3), 596–604 https://doi.org/10.1016/j.numecd.2021.11.017
82. Steyn, H., & Walt, J. (2018). Setting a cut-off score for a placement test at tertiary level. Journal for Language Teaching, 51(2), 105–118. https://doi.org/10.4314/JLT.V51I2.5
83. Su, D., Zhang, X., He, K., & Chen, Y. (2020). Use of machine learning approach to predict depression in the elderly in China: A longitudinal study. Journal of Affective Disorders, 282, 289–298. https://doi.org/10.1016/j.jad.2020.12.160
84. Şahin Sarkın, D. B. (2017). Çoklu Puanlanan Ölçeklerde Angoff, Roc ve Sınır Grup Yöntemleri ile kesme puanının belirlenmesi [Doktora Tezi, Ankara Üniversitesi]. Yüksek Öğretim Kurulu Tez Merkezi. https://tez.yok.gov.tr/UlusalTezMerkezi/
85. Şencan, H. (2005). Sosyal ve davranışsal ölçümlerde güvenilirlik ve geçerlilik (1. Baskı). Seçkin Yayıncılık.
86. Tan, S., & Zhu, S. (2023). Binary search of the optimal cut-point value in ROC analysis using the F1 score. Journal of Physics: Conference Series, 2609. https://doi.org/10.1088/1742-6596/2609/1/012002
87. Tanrıverdi, S. (2006). Standart belirleme yöntemlerinin geçme puanları üzerine etkisi (Yayımlanmamış yüksek lisans tezi). Hacettepe Üniversitesi, Sosyal Bilimler Enstitüsü, Ankara.
88. Taşdelen, G. (2009). Nedelsky ve Angoff standart belirleme yöntemlerinin genellenebilirlik kuramı ile karşılaştırılmasına ilişkin bir araştırma (Yayımlanmış yüksek lisans tezi). Hacettepe Üniversitesi, Sosyal Bilimler Enstitüsü, Ankara.
89. Taşdelen, G., Kelecioğlu, H. & Güler, N. (2010). Nedelsky ve Angoff standart belirleme yöntemleri ile elde edilen kesme puanlarının genellenebilirlik kuramı ile karşılaştırılması. Journal of Measurement and Evaluation in Education and Psychology, 1(1), 22–28.
90. Taşdemir, F., & Çokluk, Ö. (2013). Angoff (1-0), Nedelsky ve sınır değerleri saptama yöntemleri ile bir testin sınıflama doğruluklarının incelenmesi. Mediterranean Journal Of Humanities, 3(2), 241–261.
91. Terluin, B., Eekhout, I., Terwee, C. B., & de Vet, H. C. W. (2015). Minimal important change (MIC) based on a predictive modeling approach was more precise than MIC based on ROC analysis. Journal of Clinical Epidemiology, 68(12), 1388–1396. https://doi.org/10.1016/j.jclinepi.2015.03.015
92. The Jamovi Project (2021). Jamovi. (Versiyon 2.2) [Bilgisayar yazılımı]. https://www.jamovi.org.
93. Thorndike, R. L., Angoff, W. H., & Lindquist, E. F. (1971). Educational measurement (2. baskı). American Council on Education.
94. Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38(1), 1–10. https://doi.org/10.1007/BF02291170
95. Turgut, M. F., & Baykul, Y. (2014). Eğitimde ölçme ve değerlendirme (6.Baskı). Pegem Akademi.
96. Vivo, J. M., & Franco, M. (2008). How does one assess the accuracy of academic success predictors? ROC analysis applied to university entrance factors. International Journal Of Mathematical Education in Science and Technology, 398(3), 325–340.
97. Yim, M., & Shin, S. (2020). Using the Angoff method to set a standard on mock exams for the Korean Nursing Licensing Examination. Journal of Educational Evaluation for Health Professions, 17, 14. https://doi.org/10.3352/jeehp.2020.17.14
98. Yun, C., Tang, F., Gao, Z., Wang, W., Bai, F., Miller, J., Liu, H., Lee, Y., & Lou, Q. (2024). Construction of risk prediction model of type 2 diabetic kidney disease based on deep learning. Diabetes & Metabolism Journal, 48, 771–779. https://doi.org/10.4093/dmj.2023.0033
99. Yurtçu, M., & Güzeller, C. (2022). Comparison of item response theory scaling methods with ROC analysis. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 13(2), 164–183. https://doi.org/10.21031/epod.892079
100. Wang, D., & Keller, L. (2024). Using ROC analysis to refine cut scores following a standard setting process. Educational and Psychological Measurement, 85, 313–335. https://doi.org/10.1177/00131644241278925
101. Wiliam, D. (1998). Construct-referenced assessment of authentic tasks: Alternatives to norms and criteria. In Proceedings of the 24th Annual Conference of the International Association for Educational Assessment: Testing and evaluation: Confronting the challenges of rapid social change. International Association for Educational Assessment.
102. Zhang, L., Wu, J., Yu, R., Xu, R., Yang, J., Fan, Q., Wang, D., & Zhang, W. (2023). Non-contrast CT radiomics and machine learning for outcomes prediction of patients with acute ischemic stroke receiving conventional treatment. European Journal Of Radiology, 165, 110959. https://doi.org/10.1016/j.ejrad.2023.110959
103. Zhou, X. H., Castelluccio, P., & Zhou, C. (2005). Nonparametric estimation of ROC curves in the absence of a gold standard. Biometrics, 61(2), 600–609. https://doi.org/10.1111/j.1541-0420.2005.00324.x
104. Zweig, M. H. & Campbell, G. (1993) Receiver-Operating Characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine. Clinical Chemistry, 39, 561–577. https://doi.org/10.1093/clinchem/39.4.561