Evaluating ChatGPT in Generating Feedback on Content and Organization Components of EFL Compare and Contrast Essays

Abstract

ChatGPT, an innovative large language model that has impressed worldwide audiences with its exceptional generative capabilities, is now positioned to significantly transform the field of education. The purpose of this exploratory study is to investigate how accurately ChatGPT generates feedback on the content and organization components of EFL compare and contrast essays and the extent to which the feedback length provided by ChatGPT differs from that of the human teacher.To address these questions, a ChatGPT prompt incorporating evaluation criteria for content and organization components was developed, generating feedback on 10 compare and contrast student essays using the ChatGPT 3.5 version. The ChatGPT feedback and teacher feedback were assessed quantitatively and qualitatively according to the predetermined evaluation criteria. Furthermore, two types of feedback were compared descriptively and by conducting the Wilcoxon Sign Rank Test. The findings revealed that ChatGPT produced highly accurate feedback for both content and organization components, surpassing the teacher in the length of feedback provided. While the accuracy rate of the generated feedback was high, issues such as holistic assessment of the essay, false positives, failure to provide feedback where needed, and discrepancies in the depth of feedback compared to teacher feedback were identified. The results suggest that while ChatGPT shows promise in providing educational feedback, teacher-AI collaboration in giving feedback for EFL compare and contrast essays is important for delivering feedback that optimally benefits learners.

Keywords

• Artificial Intelligence
• Compare and Contrast Essay
• ChatGPT
• EFL
• Feedback

References

1. Bakla, A. (2020). A mixed-methods study of feedback modes in EFL writing. Language Learning & Technology, 24(1), 107–128. https://doi.org/10125/44712
2. Banihashem, S, K., Kerman, N.T., Noroozi, O., Moon, J., & Drachsler, H. (2024). Feedback sources in essay writing: Peergenerated or AI-generated feedback? International Journal of Educational Technology in Higher Education, 21(23), 1-15. https://doi.org/10.1186/s41239-024-00455-4
3. Barrett, A., & Pack, A. (2023). Not quite eye to AI: Student and teacher perspectives on the use of generative artificial intelligence in the writing process. International Journal of Educational Technology in Higher Education, 20(59). https://doi.org/10.1186/s41239-023-00427-0
4. Barrot, J. S. (2023a). Using ChatGPT for second language writing: Pitfalls and potentials. Assessing Writing, 57. https://doi.org/10.1016/j.asw.2023.100745
5. Barrot, J. S. (2023b). Using automated written corrective feedback in the writing classrooms: Effects on L2 writing accuracy. Computer Assisted Language Learning, 36(4), 584-607. https://doi.org/10.1080/09588221.2021.1936071
6. Baskara, F. (2023). Integrating ChatGPT into EFL writing instruction: Benefits and challenges. International Journal of Education and Learning, 5(1), 44-55. https://doi.org/10.31763/ijele.v5i1.858
7. Biber, D., Nekrasova, T., & Horn, B. (2011). The effectiveness of feedback for L1-English and L2-writing development: A meta-analysis. ETS Research Report Series, 2011(1), i-99. https://doi.org/10.1002/j.2333-8504.2011.tb02241.x
8. Boardman, C.A., & Frydenberg, J. (2002). Writing to communicate: Paragraphs and Essay (2nd ed.). Pearson Education.

9. Bok, E., & Cho, Y. (2023). Examining Korean EFL College Students’ Experiences and Perceptions of Using ChatGPT as a Writing Revision Tool. Journal of English Teaching through Movies and Media, 24(4), 15-27. https://doi.org/10.16875/stem.2023.24.4.15
10. Bonner, E., Lege, R., & Frazier, E. (2023). Large language model-based artificial intelligence in the language classroom: Practical ideas for teaching. Teaching English with Technology, 23(1), 23-41. https://doi.org/10.56297/BKAM1691/WIEO1749
11. Bozkurt, A. (2024). Tell me your prompts and I will make them true: The alchemy of prompt engineering and generative AI. Open Praxis, 16(2), 111-118. https://doi.org/10.55982/openpraxis.16.2.661
12. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
13. Buitrago, C. R., & Diaz, J. (2018). Flipping your writing lessons: Optimizing time in your EFL writing classroom. In J. Mehring & A. Leis (Eds.), Innovations in flipping the language classroom. (pp. 69–91) Springer.
14. Burke, D. M., & Pieterick, J. (2010). Giving Students Effective Written Feedback. Maidenhead: Open University Press.
15. Carlson, M., Pack, A., & Escalante, J. (2023). Utilizing OpenAI's GPT‐4 for written feedback. TESOL Journal, 759, e759. https://doi.org/10.1002/tesj.759
16. Chaka, C. (2023). Generative AI chatbots-ChatGPT versus YouChat versus Chatsonic: Use cases of selected areas of applied English language studies. International Journal of Learning, Teaching and Educational Research, 22(6), 1-19. https://doi.org/10.26803/ijlter.22.6.1
17. Dikli, S., & Bleyle, S. (2014). Automated essay scoring feedback for second language writers: How does it compare to instructor feedback? Assessing Writing, 22, 1–17. https://doi.org/10.1016/j.asw.2014.03.006
18. Elola, I., & Oskoz, A. (2016). Supporting second language writing using multimodal feedback. Foreign Language Annals, 49(1), 58-74. https://doi.org/10.1111/flan.12183
19. Fu, Q. K., Zou, D., Xie, H., & Cheng, G. (2024). A review of AWE feedback: types, learning outcomes, and implications. Computer Assisted Language Learning, 37(1–2), 179–221. https://doi.org/10.1080/09588221.2022.2033787
20. Giray, L. (2023). Prompt engineering with ChatGPT: A guide for academic writers. Annals of Biomedical Engineering, 51, 2629-2633. https://doi.org/10.1007/s10439-023-03272-4
21. Guo, K., & Wang, D. (2024). To resist it or to embrace it? Examining ChatGPT’s potential to support teacher feedback in EFL writing. Education and Information Technologies, 29, 8435-8463. https://doi.org/10.1007/s10639-023-12146-0
22. Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023). A large-scale comparison of human-written versus ChatGPT-generated essays. Scientific Reports, 13(1), 18617. https://doi.org/10.1038/s41598-023-45644-9
23. Huang, J. (2023). Engineering ChatGPT Prompts for EFL Writing Classes. International Journal of TESOL Studies, 5(4), 73-79. https://doi.org/10.58304/ijts.20230405
24. Jackson, D., Davidson, P.M., & Usher, K. (2022). Feeding Back and Feeding Forward. In: Successful Doctoral Training in Nursing and Health Sciences. Springer, Cham. https://doi.org/10.1007/978-3-030-87946-4_5
25. Köroğlu, & A. Çakır (Ed.), Fostering foreign language teaching and learning environments with contemporary technologies. (pp. 115-133). IGI Global. https://doi.org/10.4018/979-8-3693-0353-5.ch006
26. Kulhavy, R. W. (1977). Feedback in written instruction. Review of Educational Research, 47(2), 211–232. https://doi.org/10.3102/00346543047002211
27. Lee, A. V. Y. (2023). Supporting students’ generation of feedback in large-scale online course with artificial intelligence-enabled evaluation. Studies in Educational Evaluation, 77, 101250. https://doi.org/10.1016/j.stueduc.2023.101250
28. Lee, U., Jung, H., Jeon, Y., Sohn, Y., Hwang, W., Moon, J., & Kim, H. (2023). Few-shot is enough: exploring ChatGPT prompt engineering method for automatic question generation in English education. Education and Information Technologies, 1-33. https://doi.org/10.1007/s10639-023-12249-8
29. Link, S., Mehrzad, M., & Rahimi, M. (2022). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer Assisted Language Learning, 35(4), 605-634. https://doi.org/10.1080/09588221.2020.1743323
30. Lo, L. S. (2023). The CLEAR path: A framework for enhancing information literacy through prompt engineering. The Journal of Academic Librarianship, 49(4), 102720. https://doi.org/10.1016/j.acalib.2023.102720
31. Liu, C., Hou, J., Tu, Y. F., Wang, Y., & Hwang, G. J. (2021). Incorporating a reflective thinking promoting mechanism into artificial intelligence-supported English writing environments. Interactive Learning Environments, 31, 3340–3359. https://doi.org/10.1080/10494820.2021.2012812
32. Merriam-Webster. (n.d.). Get Looped in on 'Feedback' In Merriam-Webster.com dictionary. Retrieved from https://www.merriam-webster.com/wordplay/the-history-of-feedback#:~:text=Feedback%2C%20which%20began%20as%20an,of%20coming%20out%20of%20it
33. Meyer, J. G., Urbanowicz, R. J., Martin, P. C., O’Connor, K., Li, R., Peng, P. C., ..., & Moore, J. H. (2023). ChatGPT and large language models in academia: opportunities and challenges. BioData Mining, 16(1), 20. https://doi.org/10.1186/s13040-023-00339-9
34. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Sage.
35. Mizumoto, A., Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 100050. https://doi.org/10.1016/j.rmal.2023.100050
36. Mizumoto, A., Shintani, N., Sasaki, M., & Teng, M. F. (2024). Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment. Research Methods in Applied Linguistics, 3(2), 100116. https://doi.org/10.1016/j.rmal.2024.100116
37. Noroozi, O., Banihashem, S. K., Taghizadeh Kerman, N., Parvaneh Akhteh Khaneh, M., Babayi, M., Ashrafi, H., & Biemans, H. J. (2023). Gender differences in students’ argumentative essay writing, peer review performance and uptake in online learning environments. Interactive Learning Environments, 31 (10), 6302–6316. https://doi.org/10.1080/10494820.2022.2034887
38. OpenAI. (2024a). What is ChatGPT? Retrieved from https://help.openai.com/ en/articles/ 6783457- what- is- chatgpt
39. OpenAI. (2024b). Best practices for prompt engineering with the OpenAI API. Retrieved from https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-the- openai-api
40. Pack, A., & Maloney, J. (2023). Potential affordances of generative AI in language education: Demonstrations and an evaluative framework. Teaching English with Technology, 23(2), 4-24. https://doi.org/10.56297/BUKA4060/VRRO1747
41. Pfau, A., Polio, C., & Xu, Y. (2023). Exploring the potential of ChatGPT in assessing L2 writing accuracy for research purposes. Research methods in Applied Linguistics, 2(3), 100083. https://doi.org/10.1016/j.rmal.2023.100083
42. Punar Özçelik, N., & Yangın Ekşi, G. (2024). Cultivating writing skills: the role of ChatGPT as a learning assistant – a case study. Smart Learning Environments, 11(10), 1-18. https://doi.org/10.1186/s40561-024-00296-8
43. Ranalli, J. (2018). Automated written corrective feedback: How well can students make use of it?. Computer Assisted Language Learning, 31(7), 653-674. https://doi.org/10.1080/09588221.2018.1428994
44. Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1), https://doi.org/10.37074/jalt.2023.6.1.9
45. Schmidt-Fajlik, R. (2023). Chatgpt as a grammar checker for Japanese English language learners: A comparison with grammarly and prowritingaid. AsiaCALL Online Journal, 14(1), 105-119. https://doi.org/10.54855/acoj.231417
46. Schmidt, R., & Frota, S. N. (1986). Developing basic conversational ability in a second language: A case study of an adult learner of Portuguese. In R. R. Day (Ed.), Talking to learn: Conversation in second language acquisition (pp. 237-326). Newbury House
47. Shadiev, R., & Feng, Y. (2023). Using automated corrective feedback tools in language learning: A review study. Interactive Learning Environments, 1-29. https://doi.org/10.1080/10494820.2022.2153145
48. Song, C., & Song, Y. (2023). Enhancing academic writing skills and motivation: assessing the efficacy of ChatGPT in AI-assisted language learning for EFL students. Frontiers in Psychology, 14, 1260843. https://doi.org/10.3389/fpsyg.2023.1260843
49. Spasić, A. J., & Janković, D. S. (2023, June 29-July 1). Using ChatGPT standard prompt engineering techniques in lesson preparation: Role instructions and seed-word prompts [Paper presentation]. 58th International Scientific Conference on Information Communication and Energy Systems and Technologies (ICEST), Nis, Serbia. https://doi.org/10.1109/ICEST58410.2023.10187269
50. Stebbins, R. A. (2001). Exploratory research in the social sciences (Vol. 48). Sage.
51. Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C. B. (2024). Comparing the quality of human and ChatGPT feedback of students’ writing. Learning and Instruction, 91, 101894. https://doi.org/10.1016/j.learninstruc.2024.101894
52. Su, Y., Lin, Y., & Lai, C. (2023). Collaborating with ChatGPT in argumentative writing classrooms. Assessing Writing, 57, 100752. https://doi.org/10.1016/j.asw.2023.100752
53. Teng. M. F. (2024). A systematic review of ChatGPT for English as a foreign language writing: Opportunities, challenges and recommendations. International Journal of TESOL studies, 6(3), 36-57. https://doi.org/10.58304/ijts.20240304
54. Thi, N. K., & Nikolov, M. (2021). Feedback Treatments, Writing Tasks, and Accuracy Measures: A Critical Review of Research on Written Corrective Feedback. Tesl-Ej, 25(3), n3. Retrieved from https://files.eric.ed.gov/fulltext/EJ1332267.pdf
55. Üstünbaş, Ü. (2024). EFL Learners' Views About the Use of Artificial Intelligence in Giving Corrective Feedback on Writing: A Case Study. In Fostering Foreign Language Teaching and Learning Environments With Contemporary Technologies (pp. 115-133). IGI Global.
56. Wang, D., Su, J., & Yu, H. (2020). Feature extraction and analysis of natural language processing for deep learning English language. IEEE Access, 8, 46335-46345. https://doi.org/10.1109/ACCESS.2020.2974101
57. Wang, L., Chen, X., & Wang, C., Xu, L., Shadiev, R., & Li, Y. (2024). ChatGPT’s capabilities in providing feedback on undergraduate students’ argumentation: A case study. Thinking Skills and Creativity, 51, 101440. 46335-46345. https://doi.org/10.1016/j.tsc.2023.101440
58. Wilson, J., & Czik, A. (2016). Automated essay evaluation software in English Language Arts classrooms: Effects on teacher feedback, student motivation, and writing quality. Computers & Education, 100, 94–109. https://doi.org/10.1016/j.compedu.2016.05.004
59. Xiao, Y., & Zhi, Y. (2023). An Exploratory Study of EFL Learners’ Use of ChatGPT for Language Learning Tasks: Experience and Perceptions. Languages, 8(3), 212. https://doi.org/10.3390/languages8030212
60. Yan, D. (2023). Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation. Education and Information Technologies, 28, 13943–13967. https://doi.org/10.1007/s10639-023-11742-4
61. Zainurrahman, & Rojab, S. R. (2024). Examining Bing AI as a Solution to EFL Writing Feedback Challenges. PROJECT (Professional Journal of English Education), 7(2). Retrieved from https://journal.ikipsiliwangi.ac.id/index.php/project/article/view/21639
62. Zhai, N., & Ma, X. (2023). The Effectiveness of Automated Writing Evaluation on Writing Quality: A Meta-Analysis. Journal of Educational Computing Research, 61(4), 875-900. https://doi.org/10.1177/07356331221127300
63. Zhang, Z., & Hyland, K. (2018). Student engagement with teacher and automated feedback on L2 writing. Assessing Writing, 36, 90-102. https://doi.org/10.1016/j.asw.2018.02.004