AN INVESTIGATION ON THE EFFECTIVENESS OF CHATBOTS IN EVALUATING WRITING ASSIGNMENTS IN EFL CONTEXTS

Year 2024, Issue: 72, 295 - 329, 31.10.2024

Saliha Toscu

https://doi.org/10.21764/maeuefd.1425384

Abstract

This study aims to explore the effectiveness of chatbots, specifically ChatGPT, in evaluating writing assignments. 51 English as a foreign language (EFL) learners from Türkiye took part in the study voluntarily. Based on a quasi-experimental research design, the study involved dividing the participants into two non-equivalent groups. The participants in the control group received feedback on their writing from a teacher, while the experimental group was given feedback by the ChatGPT. Thus, the research indicated how Artificial Intelligence (AI)-generated feedback was influential in evaluating writing assignments compared to human-generated feedback. The feedback given by the teacher and the ChatGPT was analyzed via a content analysis, and the results showed that the ChatGPT and the teacher feedback had differences in the language style, register, content, and accuracy. Besides, students’ perceptions of the feedback were revealed through the Student Perceptions of Writing Feedback Scale of Marss (2016) and analyzed using Independent Samples t-Tests on the SPSS. The statistical tests indicated that the students’ perceptions did not vary whether the feedback was AI or human-generated. This study suggests that AI-generated feedback can match human feedback in terms of students’ perceptions. Still, the differences regarding feedback features show the strengths of a teacher and a bot for giving feedback by suggesting a more complementary role for AI technology in educational settings.

Keywords

Chatbots, ChatGPT, EFL writing, feedback, evaluation

YABANCI DİL OLARAK İNGİLİZCE BAĞLAMINDA YAZMA ÖDEVLERİNİ DEĞERLENDİRMEDE SOHBET BOTLARININ ETKINLIĞİ ÜZERİNE BİR ARAŞTIRMA

Abstract

Bu çalışma, sohbet botlarının, özellikle ChatGPT’in, yazma ödevlerini değerlendirmedeki etkinliğini ortaya çıkarmayı hedeflemektedir. Türkiye’deki özel bir okuldaki 51 yabancı dil olarak İngilizce öğrencisi çalışmaya gönüllü olarak katılmıştır. Yarı deneysel araştırma desenine dayalı olan çalışmada katılımcılar eşit olmayan iki gruba ayrılmıştır. Kontrol grubundaki katılımcıların yazma ödevlerine öğretmen, deney grubundakilere ChatGPT tarafından dönüt verilmiştir. Böylece, çalışma yapay zekâ tarafından üretilen geri bildirimin insan tarafından üretilen geri bildirimle karşılaştırıldığında yazma ödevlerini değerlendirmede etkinliğini göstermiştir. Öğretmen ve ChatGPT tarafından verilen dönüt içerik analizi yoluyla incelenmiştir ve sonuçlar ChatGPT ve öğretmen dönütü arasında dil stili, dil biçimi, içerik ve dil bilgisi doğruluğu konularında farklılıklar olduğunu göstermiştir. Ayrıca, çalışmada öğrencilerin geri bildirime yönelik algıları Marss'ın (2016) Öğrenci Yazma Geri Bildirim Algısı Ölçeği yoluyla ortaya çıkarılmıştır ve SPSS’te Bağımsız Grup t-Test kullanılarak analiz edilmiştir. İstatistiksel testler, öğrencilerin algılarının geri bildirimin yapay zekâ veya insan tarafından üretildiğinde değişmediğini göstermiştir. Bu çalışma, öğrenci algısı bakımından yapay zekâ teknolojisi ve insan geri bildiriminin eşleşebileceğini önermektedir, ancak dönütün özelliklerine ilişkin farklılıklar yapay zekâ teknolojisinin eğitim ortamlarında daha tamamlayıcı bir rol oynamasını önererek öğretmenin ve botun geri bildirim vermede güçlü yönlerini göstermektedir.

Keywords

değerlendirme, Sohbet Robotları, chatgpt, yabancı dil olarak İngilizce yazma, geri bildirim

References

• Adamopoulou, E., & Moussiades, L. (2020). Chatbots: History, technology, and applications. Machine Learning with Applications, 2, 1-18.
• Ahn, S. (2022). The effects of chatbot on grammar competence for Korean EFL college students. Journal of Digital Convergence. 20(3), 53-61.
• Algaraady, J., & Mahyoob, M. (2023). ChatGPTs capabilities in spotting and analyzing writing errors experienced by EFL learners. Arab World English Journal, 9, 3-17.
• Brophy, J. (1981). Teacher praise: A functional analysis. Review of Educational Research, 51(1), 5–32. https://doi.org/10.3102/00346543051001005
• Burstein, J., Chodorow, M., & Leacock, C. (2004). Automated essay evaluation: The criterion online writing service. AI Magazine, 25(3), 27.
• Carless, D. (2020). From teacher transmission of information to student feedback literacy: Activating the learner role in feedback processes. Active Learning in Higher Education. https://doi.org/10.1177/1469787420945845
• Çakmak, F. (2022). Chatbot-human interaction and its effects on EL students’ L2 speaking performance and anxiety. Novitas-ROYAL, 16(2), 113-131.
• Demir, S. (2023). Investigation of ChatGPT and real raters in scoring open-ended items in terms of inter-rater reliability. International Journal of Turkish Educational Studies, 11(21), 1072-1099.
• El Shazly, R. (2021). Effects of artificial intelligence on English speaking anxiety and speaking performance: A case study. Expert Systems, 38, 1-15. DOI: 10.1111/exsy.12667
• Fosnot, C. T. (1996). Enquiring teachers enquiring learners: A constructivist approach to teaching. Teachers College Press.
• Fryer, L., & Carpenter, R. (2006). Emerging technologies: Bots as language learning tools. Language Learning & Technology, 10(3), 8-14.
• Guo, K., & Wang, D. (2024). To resist it or to embrace it? Examining ChatGPT’s potent to support teacher feedback in EFL writing. Education and Information Technologies, 29, 8435-8463.
• Guo, K., Wang, J, & Chu, S. K. W. (2022). Using chatbots to scaffold EFL students’ argumentative writing. Assessing Writing, 54, 1-6.
• Halaweh, M. (2023). ChatGPT in education: Strategies for responsible implementation. Contemporary Educational Technology, 15(2), ep421. https://doi.org/10.30935/cedtech/13036
• Han, J., Yoo, H., Myung, J., Kim, M., Lee, T. Y., Ahn, S., & Oh, A. (2024, May 20-25). RECIPE4U: Student-ChatGPT interaction dataset in EFL writing education. LREC-COLING 2024, Torino, Italy.
• Harunasari, S. Y. (2022). Examining the effectiveness of AI-integrated approach in EFL writing: A case of ChatGPT. International Journal of Progressive Sciences and technologies (IJPSAT), 39(2), 357-368.
• Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. doi: 10.3102/003465430298487
• Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning – Are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38, 237-257.
• Hyland, F., & Hyland, K. (2001). Sugaring the pill: Praise and criticism in written feedback. Journal of Second Language Writing, 10, 185-212.
• Khanna, A., Pandey, B., Vashishta, K., Kalia, K., Bhale, P., & Das, T. (2015). A study of today’s A.I. through chatbots and rediscovery of machine intelligence. International Journal of U- and e-Service, Science and Technology, 8, 277–284. http: //dx.doi.org/10.14257/ijunesst.2015.8.7.28.
• Kim, N. (2018). Chatbots and Korean EFL students’ English vocabulary learning. Journal of Digital Convergence, 16(2), 1-7.
• Kovačević, D. (2023, March 15-17). Use of ChatGPT in ESP Teaching Process. 2023 22nd International Symposium INFOTEH-JAHORINA (INFOTEH), East Sarajevo, Bosnia and Herzegovina.
• Krashen, S. (1985). The input hypothesis: Issues and implications. New York: Longman.
• Long, M. H. (1996). The role of the linguistic environment in second language acquisition. In W. C. Ritchie & T. K. Bathia (Eds.) Handbook of research on second language.
• Marrs, S. A. (2016). Development of the student perceptions of writing feedback scale. Dissertation for the degree of Doctor of Philosophy. Virginia Commonwealth University. Richmond, Virginia.
• Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 1-13.
• Okonkwo, C. W., & Ade-Ibijilo, A. (2021). Chatbots applications in education: A systematic review. Computers and Education, 2, 1-10.
• Rad, H. S., Alipour, R., & Jafarpour, A. (2023). Using artificial intelligence to foster students’ writing feedback literacy, engagement, and outcome: A case of Wordtune application, Interactive Learning Environments, DOI: 10.1080/10494820.2023.2208170
• Szabó, F., & Csépes, I. (2023). Constructivism in language pedagogy. Hungarian Educational Research Journal, 13(3), 405–417.
• Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795
• Steiss, J., Tate, T., Graham, Cruz, S. J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C. B. (2024). Comparing the quality of human and ChatGPT feedback of students’ writing. Learning and Instruction, 91, 1-15.
• Swain, M. (1995). Three functions of output in second language learning. In G. Cook & B. Seidlhofer (Eds.), Principle and practice in applied linguistics: Studies in honor of William E. Rutherford (pp. 125–144). Oxford: Oxford University Press.
• Swain, M. (1985). Communicative competence: Some roles of comprehensible input and comprehensible output in its development. In S. Gass & C. Madden (Eds.), Input in second language acquisition (pp. 235–253). Rowley, MA: Newbury House.
• Tsai, C., Lİ, Y., & Brown, I. K. (2024). Impacts of ChatGPT-assisted writing for EFL English majors: Feasibility and challenges. Education and Information Technologies.
• Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
• Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2), 157-180.
• Yin, Q. & Satar, M. (2020). English as a foreign language learner interaction with chatbots: Negotiation for meaning. International Online Journal of Education and Teaching (IOJET), 7(2), 390-410.
• Zhang, B., & Mao, J. (2023). On the teaching and learning in the information age of “big data + internet?” – Some thoughts on the application of ChatGPT in teaching. In C. F. Peng, A. Asmawi, & C. Zhao (Eds.), Proceedings of the 2023 2nd International Conference on Educational Innovation and Multimedia Technology (EIMT 2023) (pp. 1005-1016). Atlantis Press.
• Zhang, Z. V. (2020). Engaging with automated writing evaluation (AWE) feedback on L2 writing: Student perceptions and revisions. Assessing Writing, 43.