İngilizce Yazmanın Değerlendirilmesinde Yapay Zekâ ve İnsan Puanlayıcıların Karşılaştırılması: Çok Yüzeyli Rasch Modeli

Abstract

Bu çalışma, İngilizce yazma becerilerinin değerlendirilmesinde üç yapay zekâ aracı (ChatGPT 3.5, Gemini, YouChat) ile insan puanlayıcılar arasındaki puanlayıcılar arası güvenirliği Çok Yüzeyli Rasch Modeli ile incelemektedir. 206 öğrenciden toplanan yazma yanıtları, Avrupa Dilleri Ortak Başvuru Çerçevesi temelli bütüncül rubrikle puanlanmış; yapay zekâ araçlarının doğruluk, kesinlik ve duyarlılık ölçütleri hesaplanarak insan yargısıyla tutarlılığı test edilmiştir. Bulgular, tüm puanlayıcılar arasında yüksek düzeyde puanlayıcılar arası güvenirlik elde edildiğini ve ChatGPT 3.5’in Gemini ve YouChat’e kıyasla daha iyi performans gösterdiğini ortaya koymaktadır. Sonuçlar, yapay zekânın hızlı geri bildirim, büyük veriyi işleme ve farklı bağlamlarda puanlama tutarlılığını artırma potansiyeline işaret etmektedir. Bununla birlikte, bağlamı doğru yorumlama ve ayrıntılı, amaca uygun geri bildirim üretme aşamalarında insan uzmanlığı kritik olduğundan, yapay zekânın insan puanlayıcıları tamamladığı hibrit yaklaşımlar önerilmektedir. Standartlaştırılmış yönergeler ve ortak istemler, insan ve yapay zekâ puanlayıcıların aynı koşullarda çalışmasını ve yöntemsel karşılaştırılabilirliği sağlamıştır.

Keywords

• puanlayıcılar arası güvenirlik
• yapay zekâ (YZ)
• Çok Yüzeyli Rasch Modeli
• İngilizce yazma becerisi
• YZ puanlaması

Artificial Intelligence Versus Human Raters in English Writing Assessment: The Multifaceted Rasch Model

Abstract

This study examines the inter-rater reliability (IRR) of three AI tools (ChatGPT 3.5, Gemini, YouChat) and four human raters in evaluating English writing skills using the multifaceted Rasch model. The evaluation focuses on the consistency and dependability of scores assigned by the AI tools compared to human judgment. The sample comprised 206 students whose responses were evaluated using a holistic scoring rubric based on the Common European Framework of Reference for Languages (CEFR) writing competencies. Findings indicate that ChatGPT 3.5 outperforms Gemini and YouChat regarding accuracy, precision, and recall, with an overall high IRR among all raters. The results suggest that AI tools can effectively supplement human raters, providing consistent and reliable assessments, which has significant implications for educational evaluations and the potential integration of AI in various scoring scenarios. The study contributes to understanding the role of AI in educational assessment and emphasizes the potential of AI tools to improve the scalability and efficiency of scoring processes. The study also discusses the importance of AI tools in providing prompt feedback, analyzing large datasets, and ensuring scoring consistency across different contexts. It also highlights the need for hybrid models that combine AI's consistency with human assessors' contextual understanding.

Keywords

• inter-rater reliability
• artificial intelligence (AI)
• Multi-faceted Rasch Model
• English language writing skill
• AI rating

References

1. Anastasi, A. (1979). Fields of applied psychology (2nd ed.). McGraw-Hill.
2. Brennan, R. L. (2001). Generalizability theory. Springer-Verlag
3. Brookhart, S.M. (2010). How to assess higher-order thinking skills in your classroom. ASCD.
4. Brookhart, S.M. (2015). Making the most of multiple choice. educational leadership, 73(1), 36 39. https://eric.ed.gov/?id=EJ1075062
5. Brooks, V. (1980) Improving the reliability of essay marking: a survey of the literature with particular reference to the English language composition. (CSE Research Project Report 5). Leicester University.
6. Burton, S. J., Sudweeks, R. R., Merrill, P. F., & Wood, B. (1991). How to prepare better multiple-choice test items: Guidelines for university faculty. Bringham Young University Testing Services and the Department of Instructional Science. https://testing.byu.edu/info/handbooks/betteritems.pdf.
7. Chen, J. (2021). Refining the teacher emotion model: Evidence from a review of literature published between 1985 and 2019. Cambridge Journal of Education, 51(3), 327–357.
8. Creswell, J. W., & Creswell, J. D. (2017). Research design: Qualitative, quantitative, and mixed methods approaches. SAGE publications.

9. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. CBS College Publishing
10. DeMars, C. E. (2010). Item response theory. Oxford University Press.
11. Doğan, N. (2013). Yazılı Yoklamalar. H. Atılgan (Ed.), Eğitimde ölçme ve değerlendirme içinde (145-168). Anı Yayıncılık.
12. Eckes, T. (2023). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments (2nd rev. ed.). Peter Lang.
13. Educational Testing Service (1987). ETS standards for quality and fairness.
14. Gerard, L. F., & Linn, M. C. (2016). Using automated scores of student essays to support teacher guidance in classroom inquiry. Journal of Science Teacher Education, 27(1), 111-129.
15. Goodwin, L.D. (2001). Interrater agreement and reliability. Measurement in Physical Education and Exercises Science, 5 (1), 13-34. http://dx.doi.org/10.1207/S15327841MPEE0501_2.
16. Gürdil, H., Soğuksu, Y. B., Salihoğlu, S., & Coşkun, F. (2025). Eğitimde ölçmede yapay zekânın entegrasyonu: Madde tepki kuramı kapsamında veri üretiminde ChatGPT’nin etkililiği. Trakya Eğitim Dergisi, 15(2), 887–918. https://doi.org/10.24315/tred.1509299
17. Gürdil, H., Anadol, H. Ö., & Soğuksu, Y. B. (2026). The use of artificial intelligence tools in assessing content validity: A comparative study with human experts. Ankara University Journal of Faculty of Educational Sciences, 59(2), 549–590. https://doi.org/10.30964/auebfd.1807153
18. Harlen, W. (2007). Assessment of Learning. SAGE Publications.
19. Harunasari, S. Y. (2023). Examining the effectiveness of AI-integrated approach in EFL writing: A case of ChatGPT. International Journal of Progressive Sciences and Technology(IJPSAT),39(2),357–368. https://ijpsat.org/index.php/ijpsat/article/download/5516/3447
20. Imran, M., & Almusharraf, N. (2023). Analyzing the role of ChatGPT as a writing assistant at higher education level: A systematic review of the literature. Contemporary Educational Technology, 15(4), ep464. https://doi.org/10.30935/cedtech/13605
21. Jackaria, P. M., Hajan, B. H., & Mastul, A. H. (2024). A comparative analysis of the rating of college students' essays by ChatGPT versus human raters. International Journal of Learning, Teaching and Educational Research, 23(2), 478-492. https://doi.org/10.26803/ijlter.23.2.23
22. Johnson, R. L., Penny, J. A., & Gordon, B. (2008). Assessing performance: Designing, scoring, and validating performance tasks. Guilford Press.
23. Jonsson A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130-144. https://doi.org/ 10.1016/j.edurev.2007.05.002.
24. Karakaya, İ. (2018). Uygulamalı ölçme ve değerlendirme. Pegem Akademi. https://doi.org/10.14527/9786052412367
25. Kim, H., Baghestani, S., Yin, S., Karatay, Y., Kurt, S., Beck, J., & Karatay, L. (2024). ChatGPT for writing evaluation: Examining the accuracy and reliability of AI-generated scores compared to human raters. In C. A. Chapelle, G. H. Beckett, & J. Ranalli (Eds.), Exploring artificial intelligence in applied linguistics (pp. 73–95). Iowa State University Digital Press. https://doi.org/10.31274/isudp.2024.154.06
26. Kollias, C. (2025). AVANT STAMP CEFR English proficiency test: Reading and listening cut score study—Technical report. Polytomous Limited / Avant Assessment. https://cdnprodwpv2.avantassessment.com/wp-content/uploads/AVANT-STAMP-for-CEFR-Cut-Scores-Technical-Report_final.pdf
27. Krippendorff, K. (2019). Content analysis: An introduction to its methodology. SAGE. https://doi.org/10.4135/9781071878781
28. Kutlu, Ö., Doğan, C. D., & Karakaya, İ. (2014). Ölçme ve değerlendirme: Performansa ve portfolyoya dayalı durum belirleme. Pegem Akademi.
29. Lamprianou, J. (2004) Marking quality assurance procedures: identifying good practice internationally. [Report prepared for the National Assessment Agency].
30. Linacre, J. M. (2002). Number of person or item strata. Rasch Measurement Transactions, 16, 888.
31. Linacre, J. M. (2004). Rasch model estimation: Further topics. Journal of Applied Measurement, 5(1), 95-110.
32. McMurtrie B (2022, December 13). AI and the future of undergraduate writing. The Chronicle of Higher Education. https://www.chronicle.com/article/ai-and-the-future-of-undergraduate-writing
33. Moskal, B. M. (2000). Scoring rubrics: What, when and how? Practical Assessment, Research & Evaluation, 7(3), 1–7. https://doi.org/10.7275/q7rm-gg74
34. O'Connell, B., De Lange, P., Freeman, M., Hancock, P., Abraham, A., Howieson, B., & Watty, K. (2016). Does calibration reduce variability in the assessment of accounting learning outcomes? Assessment & Evaluation in Higher Education, 41(3), 331–349. https://doi.org/10.1080/02602938.2015.1008398
35. Popham, W.J. (2000). Modern educational measurement: practical guidelines for educational leaders (3rd ed.). Allyn & Bacon.
36. Provost, F., & Fawcett, T. (2001). Robust classification for imprecise environments. Machine learning, 42, 203-231.
37. Sabharwal, N., & Agrawal, A. (2021). Hands-on question answering systems with BERT: Applications in neural networks and natural language processing. Apress. https://doi.org/10.1007/978-1-4842-6664-9
38. Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427-437.
39. Tabachnick, B.G., & Fidell, L.S. (2012). Using multivariate statistics (6th ed.). Allyn & Bacon.
40. Weng, X., Xia, Q., Gu, M., Rajaram, K., & Chiu, T. K. F. (2024). Assessment and learning outcomes for generative AI in higher education: A scoping review on current research status and trends. Australasian Journal of Educational Technology, 40(6), 37–55. https://doi.org/10.14742/ajet.9540
41. Woehr, D.J., & Huffuct, A.I. (1994). Rater training for performance appraisal. A quantitative review. Journal of Occupational and Organizational Psychology, 67(3), 189-205.
42. Xia, Q., Weng, X., Ouyang, F., Lin, T. J., & Chiu, T. K. F. (2024). A scoping review on how generative artificial intelligence transforms assessment in higher education. International Journal of Educational Technology in Higher Education, 21, Article 40. https://doi.org/10.1186/s41239-024-00468-z
43. Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125–145. https://doi.org/10.1177/014662168400800201
44. Zacharis, G., & Papadakis, S. (2025). Can AI grade like a human? Validity, reliability, and fairness in university coursework assessment. Educational Process: International Journal, 19, e2025591. https://doi.org/10.22521/edupij.2025.19.591