ChatGPT: Is It Reliable as an Automated Writing Evaluation Tool?

Yıl 2025, Cilt: 15 Sayı: 1, 329 - 349, 22.03.2025

Saliha Toscu

https://doi.org/10.18039/ajesi.1463503

Öz

This study primarily aims to give an understanding of whether or not teachers could rely on AI technology, specifically ChatGPT, to score students’ writings. The study was conducted with the participation of EFL university students. The students were assigned different writing tasks for five weeks, and the tasks were scored by a teacher and ChatGPT separately. Then, their scores were compared to see the extent to which ChatGPT and teacher scores differed on the SPSS. The test results indicated no statistically significant differences in the scores the bot or the teacher gave. Additionally, the results were supported by the qualitative analysis of the teachers’ perception of ChatGPT use for automated writing evaluation. The teachers’ perceptions indicated their positive attitudes towards its use for the evaluation process and general use for enhancing instruction and learning, together with the concerns and suggestions to make the most of ChatGPT. The study gives insights into the integration of ChatGPT into the assessment process and its effectiveness for class practices.

Anahtar Kelimeler

artificial intelligence, assessment, ChatGPT, writing

ChatGPT: Otomatik Bir Yazma Değerlendirme Aracı olarak Güvenilir mi?

Öz

Bu çalışma öncelikle öğretmenlerin öğrencilerin yazılarını puanlamak için yapay zeka teknolojisine, özellikle chatgpt'ye güvenip güvenemeyeceğine dair bir anlayış kazandırmayı amaçlamaktadır. Yarı deneysel bir araştırma tasarımına dayanan çalışma, EFL üniversitesi öğrencilerinin katılımıyla gerçekleştirilmiştir. Öğrencilere beş hafta boyunca farklı yazma ödevleri verildi ve ödevler bir öğretmen ve ChatGPT tarafından ayrı ayrı puanlandı. Daha sonra, ChatGPT ve bir öğretmenin puanlarının SPSS’de ne ölçüde farklı olduğunu görmek için puanları birbirleriyle karşılaştırıldı. Testlerden elde edilen sonuçlar, ChatGPT veya öğretmen tarafından verilen puanlarda istatistiksel olarak anlamlı bir farklılık göstermedi. Ek olarak, sonuçlar öğretmenlerin otomatik yazma değerlendirmesi için ChatGPT kullanımı algısının nitel analizi ile desteklenmiştir. Öğretmenlerin algıları, değerlendirme sürecinde kullanımına ve öğretimi ve öğrenmeyi geliştirmek için genel kullanıma yönelik olumlu tutumlarını birlikte göstermiştir.

Anahtar Kelimeler

yapay zeka, ChatGPT, değerlendirme, yazma

Kaynakça

• Adamopoulou, E., & Moussiades, L. (2020). Chatbots: History, technology, and applications. Machine Learning with Applications, 2, 1-18. https://doi.org/10.1016/j.mlwa.2020.100006
• Alberth. (2023). The use of ChatGPT in academic writing: A blessing or a curse in English. TEFLIN Journal, 34(2), 337-352. http://dx.doi.org/10.15639/teflinjournal.v34i2/337-352
• Algaraady, J., & Mahyoob, M. (2023). ChatGPTs capabilities in spotting and analyzing writing errors experienced by EFL learners. Arab World English Journal, 9, 3-17. https://dx.doi.org/10.24093/awej/call9.1
• Avsheniuk, N., Lutsenko, O., Svyrydiuk, T., & Seminikhyna, N. (2024). Empowering language learners’ critical thinking: Evaluating ChatGPT ‘s role in English course implementation. Arab World English Journal (AWEJ), 210-224. https://dx.doi.org/10.24093/awej/ChatGPT.14
• Ayan, A. D., & Erdemir, N. (2023). EFL teachers’ perceptions of automated written corrective feedback and Grammarly. Ahmet Keleşoğlu Eğitim Fakültesi Dergisi (AKEF), 5(3), 1183-198. https://doi.org/10.38151/akef.2023.106
• Bin-Hady, W. R. A., Al-Kadi, A., Hazaea, A., Ali, J. K. M. (2023). Exploring the dimensions of ChatGPT in English language learning: A global perspective. Library Hi Tech. https://dx.doi.org/10.1108/LHT-05-2023-0200
• Chiu, M., Hwang, G., Hsia, L., & Shyu, F. (2022). Artificial intelligence-supported art education: A deep learning-based system for promoting university students’ artwork appreciation and painting outcomes. Interactive Learning Environments. https://doi.org/10.1080/10494820.2022.2100426
• Demir, S. (2023). Investigation of ChatGPT and real raters in scoring open-ended items in terms of inter-rater reliability. International Journal of Turkish Educational Studies, 11(21), 1072-1099. https://doi.org/10.46778/goputeb.1345752
• Dilekçi, A., & Karatay, H. (2023). The effects of the 21st century skills curriculum on the development of students’ creative thinking skills. Thinking Skills and Creativity, 47, 101229. https://doi.org/10.1016/j.tsc.2022.101229
• Dikli, S. (2006). An overview of automated scoring of essays. The Journal of Technology, Learning, and Assessment, 5(1),1–35.
• Escalante, J., Pack, A., & Barrett, A. (2023). AI-generated feedback on writing: Insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education, 1-20. https://doi.org/10.1186/s41239-023-00425-2
• Evmenova, A. S., Borup, J., & Shin, J. K. (2024). Harnessing the power of generative AI to support ALL learners. TechTrends, 68, 820-831. https://doi.org/10.1007/s11528-024-00966-x
• Guo, K., & Wang, D. (2024). To resist it or to embrace it? Examining ChatGPT’s potent to support teacher feedback in EFL writing. Education and Information Technologies, 29, 8435-8463. https://doi.org/ 10.1007/s10639-023-12146-0
• Halaweh, M. (2023). ChatGPT in education: Strategies for responsible implementation. Contemporary Educational Technology, 15(2), 1-11. https://doi.org/10.30935/cedtech/13036
• Harunasari, S. Y. (2022). Examining the effectiveness of AI-integrated approach in EFL writing: A case of ChatGPT. International Journal of Progressive Sciences and Technologies (IJPSAT), 39(2), 357-368.
• Ho, P. X. P. (2024). Using ChatGPT in English language learning: A study on I.T. students’ attitudes, habits, and perceptions. International Journal of TESOL & Education, 4(1), 55-68. https://doi.org/10.54855/ijte.24414
• Hussein, M. A., Hassan, H., & Nassef, M. (2019). Automated language essay scoring systems: A literature review. Peer Computer Science, 5, 1-16. https://doi.org/10.7717/peerj-cs.208
• Kucuk, T. (2024). ChatGPT integrated grammar teaching and learning in EFL classes: A study on Tishk international university students in Erbil, Iraq. Arab World English Journal, 100-111. https://dx.doi.org/10.24093/awej/ChatGPT.6
• Latif, E., & Zhai, X. (2024). Fine-tuning ChatGPT for automatic scoring. Computers and Education, 6, 1-10. https://doi.org/10.1016/j.caeai.2024.100210
• Li, J., Link, S., & Hegelheimer, V. (2015). Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction. Journal of Second Language Writing, 27, 1-18. https://doi.org/10.1016/j.jslw.2014.10.004
• Liu, X., Zhao, A., Sun, X., Zhang, K., Kou, F., & Gai, J. (2023). The rise of ChatGPT: Unlocking its potential in education. In D. Kumar, P. Loskot, & Q. Chen (Eds.), The 3rd International Conference on Internet, Education and Information Technology (IETS 2023) (pp.1230-1236). Springer. https://doi.org/10.2991/978-94-6463-230-9_148
• Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 1-13. https://doi.org/10.1016/j.rmal.2023.100050
• Mohamed, A. M. (2024). Exploring the potential of an AI-ChatBot (ChatGPT) in enhancing English as a foreign language (EFL) teaching: Perceptions of EFL faculty members. Education and Information Technologies, 29(3), 3195-3217. https://doi.org/10.1007/s10639-023-11917-z
• Moussalli, S., & Cardoso, W. (2020). Intelligent personal assistants: Can they understand and be understood by accented L2 learners? Computer Assisted Language Learning, 33(8), 865-890. https://doi.org/10.1080/09588221.2019.1595664
• Qu, K., & Wu, X. (2024). ChatGPT as a CALL tool in language education: A study of hedonic motivation adoption models in English learning Environments. Education and Information Technologies. https://doi.org/10.1007/s10639-024-12598-y
• Ranalli, J. (2018). Automated written corrective feedback: How well can students make use of it? Computer Assisted Language Learning, 31(7), 653-674. https://doi.org/10.1080/09588221.2018.1428994
• Solak, E. (2024). Revolutionizing language learning: How ChatGPT and AI are changing the way we learn languages. International Journal of Technology in Education (IJTE), 7(2), 353-372. https://doi.org/10.46328/ijte.732
• Stevenson, M., & Phakiti, A. (2014). The effects of computer-generated feedback on the quality of writing. Assessing Writing, 19, 51–65. https://doi.org/10.1016/j.asw.2013.11.007
• Susanto, D. A., Priyolistiyanto, A., Pinandhita, F., Prabowo, A. B., & Bimo, D. S. (2024). Utilizing ChatGPT on designing English language teaching (ELT) materials in Indonesia: Opportunities and challenges. Journal of Culture, English Language, Teaching & Literature, 24(1), 157-181. https://doi.org/10.24167/celt.v24i1
• Tai, T., & Chen, H. H. (2023). The impact of Google Assistant on adolescent EFL learners’ willingness to communicate. Interactive Learning Environments, 31(3), 1485-1502. https://doi.org/10.1080/10494820.2020.1841801
• Tsai, C., Lİ, Y., & Brown, I. K. (2024). Impacts of ChatGPT-assisted writing for EFL English majors: Feasibility and challenges. Education and Information Technologies. https://doi.org/10.1007/s10639-024-12722-y
• Wang, Y., Shang, H., & Briody, P. (2012). Exploring the impact of using automated writing evaluation in English as a foreign language university students’ writing. Computer Assisted Language Learning, 26(3), 1–24. https://doi.org/10.1080/09588221.2012.655300
• Wang, X., Pang, H., Wallace, M. P., Wang, Q., & Chen, W. (2022). Learners’ perceived AI presences in AI-supported language learning: A study of AI as a humanized agent from community of inquiry. Computer Assisted Language Learning. https://doi.org/10.1080/09588221.2022.2056203
• Wang, E. L., Matsumura, L. C., Litman, D., Correnti, R., Zhang, H., Rahimi, Z., Kisa, Z., Magooda, A., Howe, E., Quintana, R. (2022). Contributions to research on automated writing scoring and feedback systems. RAND Corporation. https://doi.org/10.7249/RBA1062-1
• Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2), 157–180. https://doi.org/10.1191/1362168806lr190oa
• Wilson, J., Huang, Y., Palermo, C., Beard, G., & MacArthur, C. A. (2021). Automated feedback and automated scoring in the elementary grades: Usage, Attitudes, and associations with writing outcomes in a districtwide implementation of MI write. International Journal of Artificial Intelligence in Education, 31, 234-276. https://doi.org/10.1007/s40593-020-00236-w
• Zhang, B., & Mao, J. (2023). On the teaching and learning in the information age for “big data + Internet?” – Some thoughts on the application of ChatGPT in teaching. In C. F. Peng, A. Asmawi, & C. Zhao (Eds.), Proceedings of the 2023 2nd International Conference on Educational Innovation and Multimedia Technology (EIMT 2023) (pp.1005-1016). Atlantis Press. https://doi.org/10.2991/978-94-6463-192-0_131
• Zheng, L., Niu, J., Zhong, L., & Gyasi, J. F. (2021): The effectiveness of artificial intelligence on learning achievement and learning perception: A meta-analysis. Interactive Learning Environments. https://doi.org/10.1080/10494820.2021.2015693