Evaluating ChatGPT in Generating Feedback on Content and Organization Components of EFL Compare and Contrast Essays

Year 2024, Volume: 14 Issue: 3, 515 - 538

Amine Hatun Ataş , Behice Ceyda Cengiz , Berkan Çelik

https://doi.org/10.19126/suje.1475474

Abstract

ChatGPT, an innovative large language model that has impressed worldwide audiences with its exceptional generative capabilities, is now positioned to significantly transform the field of education. The purpose of this exploratory study is to investigate how accurately ChatGPT generates feedback on the content and organization components of EFL compare and contrast essays and the extent to which the feedback length provided by ChatGPT differs from that of the human teacher.To address these questions, a ChatGPT prompt incorporating evaluation criteria for content and organization components was developed, generating feedback on 10 compare and contrast student essays using the ChatGPT 3.5 version. The ChatGPT feedback and teacher feedback were assessed quantitatively and qualitatively according to the predetermined evaluation criteria. Furthermore, two types of feedback were compared descriptively and by conducting the Wilcoxon Sign Rank Test. The findings revealed that ChatGPT produced highly accurate feedback for both content and organization components, surpassing the teacher in the length of feedback provided. While the accuracy rate of the generated feedback was high, issues such as holistic assessment of the essay, false positives, failure to provide feedback where needed, and discrepancies in the depth of feedback compared to teacher feedback were identified. The results suggest that while ChatGPT shows promise in providing educational feedback, teacher-AI collaboration in giving feedback for EFL compare and contrast essays is important for delivering feedback that optimally benefits learners.

Keywords

Artificial Intelligence, Compare and Contrast Essay, ChatGPT, EFL, Feedback

Ethical Statement

Ethical consent was sought from the Human Research Ethics Committee of Zonguldak Bülent Ecevit University (protocol number: 604).

Supporting Institution

No grants were received from any public, private or non-profit organizations for this research.

Thanks

The authors would like to express their sincere thanks to the editor and the anonymous reviewers for their helpful comments and suggestions.

References

• Bakla, A. (2020). A mixed-methods study of feedback modes in EFL writing. Language Learning & Technology, 24(1), 107–128. https://doi.org/10125/44712
• Banihashem, S, K., Kerman, N.T., Noroozi, O., Moon, J., & Drachsler, H. (2024). Feedback sources in essay writing: Peergenerated or AI-generated feedback? International Journal of Educational Technology in Higher Education, 21(23), 1-15. https://doi.org/10.1186/s41239-024-00455-4
• Barrett, A., & Pack, A. (2023). Not quite eye to AI: Student and teacher perspectives on the use of generative artificial intelligence in the writing process. International Journal of Educational Technology in Higher Education, 20(59). https://doi.org/10.1186/s41239-023-00427-0
• Barrot, J. S. (2023a). Using ChatGPT for second language writing: Pitfalls and potentials. Assessing Writing, 57. https://doi.org/10.1016/j.asw.2023.100745
• Barrot, J. S. (2023b). Using automated written corrective feedback in the writing classrooms: Effects on L2 writing accuracy. Computer Assisted Language Learning, 36(4), 584-607. https://doi.org/10.1080/09588221.2021.1936071
• Baskara, F. (2023). Integrating ChatGPT into EFL writing instruction: Benefits and challenges. International Journal of Education and Learning, 5(1), 44-55. https://doi.org/10.31763/ijele.v5i1.858
• Biber, D., Nekrasova, T., & Horn, B. (2011). The effectiveness of feedback for L1-English and L2-writing development: A meta-analysis. ETS Research Report Series, 2011(1), i-99. https://doi.org/10.1002/j.2333-8504.2011.tb02241.x
• Boardman, C.A., & Frydenberg, J. (2002). Writing to communicate: Paragraphs and Essay (2nd ed.). Pearson Education.
• Bok, E., & Cho, Y. (2023). Examining Korean EFL College Students’ Experiences and Perceptions of Using ChatGPT as a Writing Revision Tool. Journal of English Teaching through Movies and Media, 24(4), 15-27. https://doi.org/10.16875/stem.2023.24.4.15
• Bonner, E., Lege, R., & Frazier, E. (2023). Large language model-based artificial intelligence in the language classroom: Practical ideas for teaching. Teaching English with Technology, 23(1), 23-41. https://doi.org/10.56297/BKAM1691/WIEO1749
• Bozkurt, A. (2024). Tell me your prompts and I will make them true: The alchemy of prompt engineering and generative AI. Open Praxis, 16(2), 111-118. https://doi.org/10.55982/openpraxis.16.2.661
• Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
• Buitrago, C. R., & Diaz, J. (2018). Flipping your writing lessons: Optimizing time in your EFL writing classroom. In J. Mehring & A. Leis (Eds.), Innovations in flipping the language classroom. (pp. 69–91) Springer.
• Burke, D. M., & Pieterick, J. (2010). Giving Students Effective Written Feedback. Maidenhead: Open University Press.
• Carlson, M., Pack, A., & Escalante, J. (2023). Utilizing OpenAI's GPT‐4 for written feedback. TESOL Journal, 759, e759. https://doi.org/10.1002/tesj.759
• Chaka, C. (2023). Generative AI chatbots-ChatGPT versus YouChat versus Chatsonic: Use cases of selected areas of applied English language studies. International Journal of Learning, Teaching and Educational Research, 22(6), 1-19. https://doi.org/10.26803/ijlter.22.6.1
• Dikli, S., & Bleyle, S. (2014). Automated essay scoring feedback for second language writers: How does it compare to instructor feedback? Assessing Writing, 22, 1–17. https://doi.org/10.1016/j.asw.2014.03.006
• Elola, I., & Oskoz, A. (2016). Supporting second language writing using multimodal feedback. Foreign Language Annals, 49(1), 58-74. https://doi.org/10.1111/flan.12183
• Fu, Q. K., Zou, D., Xie, H., & Cheng, G. (2024). A review of AWE feedback: types, learning outcomes, and implications. Computer Assisted Language Learning, 37(1–2), 179–221. https://doi.org/10.1080/09588221.2022.2033787
• Giray, L. (2023). Prompt engineering with ChatGPT: A guide for academic writers. Annals of Biomedical Engineering, 51, 2629-2633. https://doi.org/10.1007/s10439-023-03272-4
• Guo, K., & Wang, D. (2024). To resist it or to embrace it? Examining ChatGPT’s potential to support teacher feedback in EFL writing. Education and Information Technologies, 29, 8435-8463. https://doi.org/10.1007/s10639-023-12146-0
• Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023). A large-scale comparison of human-written versus ChatGPT-generated essays. Scientific Reports, 13(1), 18617. https://doi.org/10.1038/s41598-023-45644-9
• Huang, J. (2023). Engineering ChatGPT Prompts for EFL Writing Classes. International Journal of TESOL Studies, 5(4), 73-79. https://doi.org/10.58304/ijts.20230405
• Jackson, D., Davidson, P.M., & Usher, K. (2022). Feeding Back and Feeding Forward. In: Successful Doctoral Training in Nursing and Health Sciences. Springer, Cham. https://doi.org/10.1007/978-3-030-87946-4_5
• Köroğlu, & A. Çakır (Ed.), Fostering foreign language teaching and learning environments with contemporary technologies. (pp. 115-133). IGI Global. https://doi.org/10.4018/979-8-3693-0353-5.ch006
• Kulhavy, R. W. (1977). Feedback in written instruction. Review of Educational Research, 47(2), 211–232. https://doi.org/10.3102/00346543047002211
• Lee, A. V. Y. (2023). Supporting students’ generation of feedback in large-scale online course with artificial intelligence-enabled evaluation. Studies in Educational Evaluation, 77, 101250. https://doi.org/10.1016/j.stueduc.2023.101250
• Lee, U., Jung, H., Jeon, Y., Sohn, Y., Hwang, W., Moon, J., & Kim, H. (2023). Few-shot is enough: exploring ChatGPT prompt engineering method for automatic question generation in English education. Education and Information Technologies, 1-33. https://doi.org/10.1007/s10639-023-12249-8
• Link, S., Mehrzad, M., & Rahimi, M. (2022). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer Assisted Language Learning, 35(4), 605-634. https://doi.org/10.1080/09588221.2020.1743323
• Lo, L. S. (2023). The CLEAR path: A framework for enhancing information literacy through prompt engineering. The Journal of Academic Librarianship, 49(4), 102720. https://doi.org/10.1016/j.acalib.2023.102720
• Liu, C., Hou, J., Tu, Y. F., Wang, Y., & Hwang, G. J. (2021). Incorporating a reflective thinking promoting mechanism into artificial intelligence-supported English writing environments. Interactive Learning Environments, 31, 3340–3359. https://doi.org/10.1080/10494820.2021.2012812
• Merriam-Webster. (n.d.). Get Looped in on 'Feedback' In Merriam-Webster.com dictionary. Retrieved from https://www.merriam-webster.com/wordplay/the-history-of-feedback#:~:text=Feedback%2C%20which%20began%20as%20an,of%20coming%20out%20of%20it
• Meyer, J. G., Urbanowicz, R. J., Martin, P. C., O’Connor, K., Li, R., Peng, P. C., ..., & Moore, J. H. (2023). ChatGPT and large language models in academia: opportunities and challenges. BioData Mining, 16(1), 20. https://doi.org/10.1186/s13040-023-00339-9
• Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Sage.
• Mizumoto, A., Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 100050. https://doi.org/10.1016/j.rmal.2023.100050
• Mizumoto, A., Shintani, N., Sasaki, M., & Teng, M. F. (2024). Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment. Research Methods in Applied Linguistics, 3(2), 100116. https://doi.org/10.1016/j.rmal.2024.100116
• Noroozi, O., Banihashem, S. K., Taghizadeh Kerman, N., Parvaneh Akhteh Khaneh, M., Babayi, M., Ashrafi, H., & Biemans, H. J. (2023). Gender differences in students’ argumentative essay writing, peer review performance and uptake in online learning environments. Interactive Learning Environments, 31 (10), 6302–6316. https://doi.org/10.1080/10494820.2022.2034887
• OpenAI. (2024a). What is ChatGPT? Retrieved from https://help.openai.com/ en/articles/ 6783457- what- is- chatgpt
• OpenAI. (2024b). Best practices for prompt engineering with the OpenAI API. Retrieved from https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-the- openai-api
• Pack, A., & Maloney, J. (2023). Potential affordances of generative AI in language education: Demonstrations and an evaluative framework. Teaching English with Technology, 23(2), 4-24. https://doi.org/10.56297/BUKA4060/VRRO1747
• Pfau, A., Polio, C., & Xu, Y. (2023). Exploring the potential of ChatGPT in assessing L2 writing accuracy for research purposes. Research methods in Applied Linguistics, 2(3), 100083. https://doi.org/10.1016/j.rmal.2023.100083
• Punar Özçelik, N., & Yangın Ekşi, G. (2024). Cultivating writing skills: the role of ChatGPT as a learning assistant – a case study. Smart Learning Environments, 11(10), 1-18. https://doi.org/10.1186/s40561-024-00296-8
• Ranalli, J. (2018). Automated written corrective feedback: How well can students make use of it?. Computer Assisted Language Learning, 31(7), 653-674. https://doi.org/10.1080/09588221.2018.1428994
• Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1), https://doi.org/10.37074/jalt.2023.6.1.9
• Schmidt-Fajlik, R. (2023). Chatgpt as a grammar checker for Japanese English language learners: A comparison with grammarly and prowritingaid. AsiaCALL Online Journal, 14(1), 105-119. https://doi.org/10.54855/acoj.231417
• Schmidt, R., & Frota, S. N. (1986). Developing basic conversational ability in a second language: A case study of an adult learner of Portuguese. In R. R. Day (Ed.), Talking to learn: Conversation in second language acquisition (pp. 237-326). Newbury House
• Shadiev, R., & Feng, Y. (2023). Using automated corrective feedback tools in language learning: A review study. Interactive Learning Environments, 1-29. https://doi.org/10.1080/10494820.2022.2153145
• Song, C., & Song, Y. (2023). Enhancing academic writing skills and motivation: assessing the efficacy of ChatGPT in AI-assisted language learning for EFL students. Frontiers in Psychology, 14, 1260843. https://doi.org/10.3389/fpsyg.2023.1260843
• Spasić, A. J., & Janković, D. S. (2023, June 29-July 1). Using ChatGPT standard prompt engineering techniques in lesson preparation: Role instructions and seed-word prompts [Paper presentation]. 58th International Scientific Conference on Information Communication and Energy Systems and Technologies (ICEST), Nis, Serbia. https://doi.org/10.1109/ICEST58410.2023.10187269
• Stebbins, R. A. (2001). Exploratory research in the social sciences (Vol. 48). Sage.
• Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C. B. (2024). Comparing the quality of human and ChatGPT feedback of students’ writing. Learning and Instruction, 91, 101894. https://doi.org/10.1016/j.learninstruc.2024.101894
• Su, Y., Lin, Y., & Lai, C. (2023). Collaborating with ChatGPT in argumentative writing classrooms. Assessing Writing, 57, 100752. https://doi.org/10.1016/j.asw.2023.100752
• Teng. M. F. (2024). A systematic review of ChatGPT for English as a foreign language writing: Opportunities, challenges and recommendations. International Journal of TESOL studies, 6(3), 36-57. https://doi.org/10.58304/ijts.20240304
• Thi, N. K., & Nikolov, M. (2021). Feedback Treatments, Writing Tasks, and Accuracy Measures: A Critical Review of Research on Written Corrective Feedback. Tesl-Ej, 25(3), n3. Retrieved from https://files.eric.ed.gov/fulltext/EJ1332267.pdf
• Üstünbaş, Ü. (2024). EFL Learners' Views About the Use of Artificial Intelligence in Giving Corrective Feedback on Writing: A Case Study. In Fostering Foreign Language Teaching and Learning Environments With Contemporary Technologies (pp. 115-133). IGI Global.
• Wang, D., Su, J., & Yu, H. (2020). Feature extraction and analysis of natural language processing for deep learning English language. IEEE Access, 8, 46335-46345. https://doi.org/10.1109/ACCESS.2020.2974101
• Wang, L., Chen, X., & Wang, C., Xu, L., Shadiev, R., & Li, Y. (2024). ChatGPT’s capabilities in providing feedback on undergraduate students’ argumentation: A case study. Thinking Skills and Creativity, 51, 101440. 46335-46345. https://doi.org/10.1016/j.tsc.2023.101440
• Wilson, J., & Czik, A. (2016). Automated essay evaluation software in English Language Arts classrooms: Effects on teacher feedback, student motivation, and writing quality. Computers & Education, 100, 94–109. https://doi.org/10.1016/j.compedu.2016.05.004
• Xiao, Y., & Zhi, Y. (2023). An Exploratory Study of EFL Learners’ Use of ChatGPT for Language Learning Tasks: Experience and Perceptions. Languages, 8(3), 212. https://doi.org/10.3390/languages8030212
• Yan, D. (2023). Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation. Education and Information Technologies, 28, 13943–13967. https://doi.org/10.1007/s10639-023-11742-4
• Zainurrahman, & Rojab, S. R. (2024). Examining Bing AI as a Solution to EFL Writing Feedback Challenges. PROJECT (Professional Journal of English Education), 7(2). Retrieved from https://journal.ikipsiliwangi.ac.id/index.php/project/article/view/21639
• Zhai, N., & Ma, X. (2023). The Effectiveness of Automated Writing Evaluation on Writing Quality: A Meta-Analysis. Journal of Educational Computing Research, 61(4), 875-900. https://doi.org/10.1177/07356331221127300
• Zhang, Z., & Hyland, K. (2018). Student engagement with teacher and automated feedback on L2 writing. Assessing Writing, 36, 90-102. https://doi.org/10.1016/j.asw.2018.02.004