Araştırma Makalesi

Evaluating Human Reviewers versus Artificial Intelligence: A Comparative Analysis of AI-Generated Essay Detection in Higher Education

Sayı: 67 21 Ağustos 2026
PDF İndir
EN TR

Evaluating Human Reviewers versus Artificial Intelligence: A Comparative Analysis of AI-Generated Essay Detection in Higher Education

Öz

The proliferation of AI text generation tools in educational settings has created unprecedented challenges for academic integrity. As AI-generated content grows more sophisticated, traditional assessment methods may no longer suffice to distinguish human from machine-authored work. This mixed-methods study compared human reviewers, an AI scoring model, and plagiarism detection tools in identifying AI-generated essays. Sixteen essays were analyzed: eight by ELT master's students and eight AI-generated (ChatGPT 4o, Gemini 1.5, Llama), with four subsequently humanized. ELT was selected because writing proficiency and authentic voice are central disciplinary competencies. Two faculty reviewers, iThenticate, Turnitin, and a GPT-4o model evaluated all essays using an expert-reviewed analytic rubric covering coherence, originality, depth, writing style, and topic adherence. The AI model showed superior internal consistency (α = 0.923) versus human reviewers (α = 0.832). Turnitin achieved the highest accuracy (87.5%). Human reviewers performed inconsistently (62.5%; Fisher's Exact Test, p = .096). Inter-rater agreement was assessed via quadratic weighted kappa (QWK), as small sample sizes precluded a valid chi-square analysis. Digital tools currently outperform human judgment in detecting AI-generated content, yet limitations persist across all methods. These findings underscore the need for institutional AI policies, educator training, and hybrid human-technological assessment approaches.

Anahtar Kelimeler

AI-generated text detection, academic integrity, English language teaching, automated essay evaluation

Etik Beyan

This research was conducted with the permission obtained by the Tokat Gaziosmanpaşa University Scientific Research and Publication Ethics Social and Human Sciences Board's decision dated 23/07/2024 and numbered 01-33.

Kaynakça

  1. Alzubi, A. A. F., Nazim, M., & Alyami, N. (2025). Do AI-generative tools kill or nurture creativity in EFL teaching and learning? Education and Information Technologies, 30, 15147 15184. https://doi.org/10.1007/s10639-025-13409-8
  2. Atkinson, R. C., & Shiffrin, R. M. (1968). Human memory: A proposed system and its control processes. Psychology of Learning and Motivation, 2, 89-195. Academic Press.
  3. Ayub, T., Ahmad Malla, R., Khan, M. Y., & Ganaie, S. A. (2024). The art of deception: Humanizing AI to outsmart detection. Global Knowledge, Memory and Communication, 73(8), 889 905. https://doi.org/10.1108/GKMC-03-2024-0133
  4. Bearman, M., & Luckin, R. (2020). Preparing university assessment for a world with AI: Tasks for human intelligence. In M. Bearman, P. Dawson, R. Ajjawi, J. Tai, & D. Boud (Eds.), Re-imagining university assessment in a digital world (pp. 49-63). Springer. https://doi.org/10.1007/978-3-030-41956-1_5
  5. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
  6. Chein, J. M., Martinez, S. A., & Barone, A. R. (2024). Human intelligence can safeguard against artificial intelligence: Individual differences in the discernment of human from AI texts. Scientific Reports, 14(1), Article 25989. https://doi.org/10.1038/s41598-024-76218-y
  7. Chung, J. Y., & Jeong, S.-H. (2024). Exploring the perceptions of Chinese pre-service teachers on the integration of generative AI in English language teaching: Benefits, challenges, and educational implications. Online Journal of Communication and Media Technologies, 14(4), e202457. https://doi.org/10.30935/ojcmt/15266
  8. Cingillioglu, I. (2023). Detecting AI-generated essays: The ChatGPT challenge. The International Journal of Information and Learning Technology, 40(4), 259 268. https://doi.org/10.1108/IJILT-03-2023-0043
  9. Coccoli, M., & Patanè, G. (2024). AI vs. AI: The detection game. In 2024 IEEE 8th Forum on Research and Technologies for Society and Industry Innovation (RTSI) (pp. 1 6). IEEE. https://doi.org/10.1109/RTSI61910.2024.10761124
  10. Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98–104. https://doi.org/10.1037/0021-9010.78.1.98

Kaynak Göster

APA
Susoy, Z. (2026). Evaluating Human Reviewers versus Artificial Intelligence: A Comparative Analysis of AI-Generated Essay Detection in Higher Education. Pamukkale Üniversitesi Eğitim Fakültesi Dergisi, 67, 342-358. https://doi.org/10.9779/pauefd.1847553
AMA
1.Susoy Z. Evaluating Human Reviewers versus Artificial Intelligence: A Comparative Analysis of AI-Generated Essay Detection in Higher Education. PAÜEFD. 2026;(67):342-358. doi:10.9779/pauefd.1847553
Chicago
Susoy, Zafer. 2026. “Evaluating Human Reviewers versus Artificial Intelligence: A Comparative Analysis of AI-Generated Essay Detection in Higher Education”. Pamukkale Üniversitesi Eğitim Fakültesi Dergisi, sy 67: 342-58. https://doi.org/10.9779/pauefd.1847553.
EndNote
Susoy Z (01 Ağustos 2026) Evaluating Human Reviewers versus Artificial Intelligence: A Comparative Analysis of AI-Generated Essay Detection in Higher Education. Pamukkale Üniversitesi Eğitim Fakültesi Dergisi 67 342–358.
IEEE
[1]Z. Susoy, “Evaluating Human Reviewers versus Artificial Intelligence: A Comparative Analysis of AI-Generated Essay Detection in Higher Education”, PAÜEFD, sy 67, ss. 342–358, Ağu. 2026, doi: 10.9779/pauefd.1847553.
ISNAD
Susoy, Zafer. “Evaluating Human Reviewers versus Artificial Intelligence: A Comparative Analysis of AI-Generated Essay Detection in Higher Education”. Pamukkale Üniversitesi Eğitim Fakültesi Dergisi. 67 (01 Ağustos 2026): 342-358. https://doi.org/10.9779/pauefd.1847553.
JAMA
1.Susoy Z. Evaluating Human Reviewers versus Artificial Intelligence: A Comparative Analysis of AI-Generated Essay Detection in Higher Education. PAÜEFD. 2026;:342–358.
MLA
Susoy, Zafer. “Evaluating Human Reviewers versus Artificial Intelligence: A Comparative Analysis of AI-Generated Essay Detection in Higher Education”. Pamukkale Üniversitesi Eğitim Fakültesi Dergisi, sy 67, Ağustos 2026, ss. 342-58, doi:10.9779/pauefd.1847553.
Vancouver
1.Zafer Susoy. Evaluating Human Reviewers versus Artificial Intelligence: A Comparative Analysis of AI-Generated Essay Detection in Higher Education. PAÜEFD. 01 Ağustos 2026;(67):342-58. doi:10.9779/pauefd.1847553