ChatGPT-4o as an automated scoring tool for writing assessment: Strengths and weaknesses

Abstract

ChatGPT is widely used for many educational purposes such as content generation and language translation, however, its role as an automated scoring tool requires further empirical investigation. This mixed-method study explores the effectiveness of ChatGPT-4o as an automated scoring tool for English as a Foreign Language (EFL) learners’ written output. It particularly aims to discover to what extent ChatGPT-4o can produce reliable and accurate scores in writing assessment and whether it can serve as an alternative scoring tool to the traditional human scoring. 240 argumentative essays were first scored by 13 human raters working in pairs. 28 of them were selected as model essays while the remaining 212 essays were thereafter scored by ChatGPT-4o only. Quantitative analysis employed the Quadratic Weighted Kappa statistic to measure inter-rater reliability, focusing on the agreement among human raters and ChatGPT-4o. Findings suggest that ChatGPT-4o demonstrates only fair agreement with human raters, producing significantly lower and inconsistent scores. Regarding this discrepancy, five experienced human raters were interviewed about the strengths and weaknesses of ChatGPT as a scoring tool, with their perspectives and practices thematically analyzed to triangulate the quantitative findings. The key differences were classified under the themes such as rubric adherence, scoring bias and sensitivity to nuances. Due to AI-enabled automation, ChatGPT exhibits pragmatic dualities in practicality, providing feedback and linguistic capacity. The remarkable strengths involve less manual effort, faster detailed scoring feedback and broader linguistic dataset. However, human-driven optimization through constant supervision, care and pedagogical expertise is essential for a more nuanced scoring.

Keywords

• Automated scoring tool
• ChatGPT-4o
• Writing assessment

References

1. Aliakbari, M., Barzan, P., & Sayyadi, M. (2025). Automated feedback vs. Human feedback: A study on AI-Driven language assessment. AI and Tech in Behavioral and Social Sciences, 3(2), 113-126. https://doi.org/10.61838/kman.aitech.3.2.9
2. Attali, Y., & Burstein, J. (2005). Automated essay scoring with e-rater® V.2.0. ETS Research Report Series, 2004(2), i–21. https://doi.org/10.1002/j.2333-8504.2004.tb01972.x
3. Attali, Y., Bridgeman, B., & Trapani, C. (2010). Performance of a generic approach in automated essay scoring. Journal of Technology, Learning and Assessment, 10(3), 1 17. https://ejournals.bc.edu/index.php/jtla/article/view/1603
4. Barrot, J.S. (2023). Using ChatGPT for second language writing: Pitfalls and potentials. Assessing Writing, 57, Article 100745, 1-6. https://doi.org/10.1016/j.asw.2023.100745
5. Bandura, A. (2002). Growing primacy of human agency in adaptation and change in the electronic era. European Psychologist, 7(1), 2-16. https://doi.org/10.1027/1016-9040.7.1.2
6. Bilican Demir, S., & Yıldırım, Ö. (2019). Yazılı anlatım becerilerinin değerlendirilmesi için dereceli puanlama anahtarı geliştirme çalışması [Development of an analytical rubric for assessing writing skills]. Pamukkale University Journal of Education, 47, 457 473. https://doi.org/10.9779/pauefd.588565
7. Bui, N.M., & Barrot, J.S. (2024). ChatGPT as an automated essay scoring tool in the writing classrooms: how it compares with human scoring. Education and Information Technologies, 30, 1 18. https://doi.org/10.1007/s10639-024-12891-w
8. Chapelle, C., Cotos, E., & Lee, J. (2015). Validity argument for diagnostic assessment using automated writing evaluations. Language Testing, 32(3), 385 405. https://doi.org/10.1177/0265532214565386

9. Chen, X., Zhou, Z., & Prado, M. (2025). ChatGPT-3.5 as an automatic scoring system and feedback provider in IELTS exams. International Journal of Assessment Tools in Education, 12(1), 62-77. https://doi.org/10.21449/ijate.1496193
10. Clarke, V., & Braun, V. (2016). Thematic analysis. The Journal of Positive Psychology, 12(3), 297-298. https://doi.org/10.1080/17439760.2016.1262613
11. Creswell, J.W., & Creswell, J.D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Sage Publications.
12. Creswell J.W., & Plano Clark V.L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage Publications.
13. Crossley, S.A. (2020). Linguistic features in writing quality and development: An overview. Journal of Writing Research, 11(3), 415-443. https://doi.org/10.17239/jowr-2020.11.03.01
14. Curran, P.J., West, S.G., & Finch, J.F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16 29. https://doi.org/10.1037/1082-989X.1.1.16
15. Demir, S. (2023). Investigation of ChatGPT and real raters in scoring open-ended items in terms of inter-rater reliability. International Journal of Turkish Educational Sciences, 11(21), 1072-1099. https://doi.org/10.46778/goputeb.1345752
16. Dikli, S. (2006). An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment, 5(1), 1-35. https://files.eric.ed.gov/fulltext/EJ843855.pdf
17. Dörnyei, Z. (2007). Research methods in applied linguistics. Oxford University Press.
18. Escalante, J., Pack, A., & Barrett, A. (2023). AI-generated feedback on writing: Insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education, 20(1), 57. https://doi.org/10.1186/s41239-023-00425-2
19. Fang, T., Yang, S., Lan, K., Wong, D.F., Hu, J., Chao, L.S., & Zhang, Y. (2023). Is ChatGPT a highly fluent grammatical error correction system? A comprehensive evaluation. arXiv preprint arXiv:2304.01746. https://doi.org/10.48550/arXiv.2304.01746
20. Foltz, P.W., Streeter, L.A., Lochbaum, K.E., & Landuer, T.K. (2013). Implementation and applications of the Intelligent Essay Assessor. In M.D. Shermis & J. Burnstein (Eds.), Handbook of automated essay evaluation: Current applications and new directions (pp. 68–88). Routledge.
21. Geçkin, V., Kızıltaş, E., & Çınar, Ç. (2023). Assessing second language academic writing: AI vs. Human raters. Journal of Educational Technology & Online Learning, 6(4), 1096 1108. https://doi.org/10.31681/jetol.1336599
22. Gill, G.S., Blair, J., & Litinsky, S. (2024) Evaluating the performance of ChatGPT 3.5 and 4.0 on StatPearls oculoplastic surgery text and image based exam questions. Cureus, 16(11). https://doi.org/10.7759/cureus.73812
23. Glazko, K., Mohammed, Y., Kosa, B., Potluri, V., & Mankoff, J. (2024, June). Identifying and improving disability bias in GPT-based resume screening. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (pp. 687 700). Association for Computing Machinery. https://doi.org/10.1145/3630106.3658933
24. Guo, K., & Wang, D. (2023). To resist it or to embrace it? Examining ChatGPT’s potential to support teacher feedback in EFL writing. Educ Inf Technol, 29, 8435-8463. https://doi.org/10.1007/s10639-023-12146-0
25. Huffman, S. (2015). Exploring learner perceptions of and interaction behaviors using the Research Writing Tutor for research article Introduction section draft analysis [Unpublished doctoral dissertation]. Iowa State University.
26. Hyland, K., & Hyland, F. (2006). Feedback on second language students’ writing. Language Teaching, 39(2), 83-101. https://doi.org/10.1017/S0261444806003399
27. Jukiewicz, M. (2024). The future of grading programming assignments in education: The role of ChatGPT in automating the assessment and feedback process. Thinking Skills and Creativity, 52, Article 101522. https://doi.org/10.1016/j.tsc.2024.101522
28. Kostka, I., & Toncelli, R. (2023). Exploring applications of ChatGPT to English language teaching: Opportunities, challenges, and recommendations. The Electronic Journal for English as a Second Language, 27(3), 1-19. https://doi.org/10.55593/ej.27107int
29. Landis, J.R., & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-176. https://doi.org/10.2307/2529310
30. Li, Z., Link, S., Ma, H., Yang, H. & Hegelheimer, V. (2014). The role of automated writing evaluation holistic scores in the ESL classroom. System, 44(1), 66 78. https://doi.org/10.1016/j.system.2014.02.007
31. Li, J., Zong, H., Wu, E., Wu, R., Peng, Z., Zhao, J., Yang, L., Xie H., & Shen, B. (2024). Exploring the potential of artificial intelligence to enhance the writing of English academic papers by non-native English-speaking medical students - the educational application of ChatGPT. BMC Medical Education, 24, Article 736. https://doi.org/10.1186/s12909-024-05738-y
32. Lo, C.K., Yu, P.L.H., Xu, S., Ng, D.T.K, & Jong, M.S. (2024). Exploring the application of ChatGPT in ESL/EFL education and related research issues: a systematic review of empirical studies. Smart Learning Environments, 11(1), Article 50, https://doi.org/10.1186/s40561-024-00342-5
33. Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), Article 100050. https://doi.org/10.1016/j.rmal.2023.100050
34. Mollick, E., & Mollick, L. (2023). Assigning AI: Seven approaches for students, with prompts. arXiv preprint arXiv:2306.10052. https://doi.org/10.48550/arXiv.2306.10052
35. Nguyen, P., & Hegelheimer, V. (2022). Technology and assessment. In N. Ziegler & M. González-Lloret (Eds.), The Routledge handbook of second language acquisition and technology (pp. 107-118). Routledge.
36. OpenAI (2022, November 30). ChatGPT: Optimizing language models for dialogue. OpenAI. https://openai.com/blog/chatgpt/
37. Page, E.B. (2003). Project Essay Grade: PEG. In M.D. Shermis & J. Burstein (Eds.). Automated essay scoring: A cross-disciplinary perspective, (pp. 39-50). Lawrence Erlbaum Associates.
38. Palinkas, L.A., Horwitz, S.M., Green, C.A., Wisdom, J.P., Duan, N., & Hoagwood, K. (2013). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 42(5), 533–544. https://doi.org/10.1007/s10488-013-0528-y
39. Patton, M.Q. (2002). Qualitative research and evaluation methods (3rd ed.). Sage Publications.
40. Quah, B., Zheng, L., Sng, T.J.H., Yong C.W., & Islam, I. (2024). Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations. BMC Medical Education, 24, Article 962. https://doi.org/10.1186/s12909-024-05881-6
41. Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre training. OpenAI. https://cdn.openai.com/research covers/language unsupervised/language_understanding_paper.pdf
42. Shen, X., Chen, Z., Backes, M., & Zhang, Y. (2023). In ChatGPT we trust? Measuring and characterizing the reliability of ChatGPT. arXiv preprint arXiv:2304.08979. https://doi.org/10.48550/arXiv.2304.08979
43. Shermis, M.D. (2024). Using ChatGPT to score essays and short-form constructed responses. arXiv preprint arXiv:2408.09540. https://doi.org/10.48550/arXiv.2408.09540
44. Shermis, M.D., & Burstein, J. (2013). Handbook of Automated Essay Evaluation: Current applications and new directions. (1st ed.). Routledge. https://doi.org/10.4324/9780203122761
45. Sihite, M.R., Meisuri, M., & Sibarani, B. (2023). Examining the validity and reliability of ChatGPT 3.5-generated reading comprehension questions for academic texts. Randwick International of Education and Linguistics Science Journal, 4(4), 937-944. https://doi.org/10.47175/rielsj.v4i4.835
46. Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C.B. (2024). Comparing the quality of human and ChatGPT feedback of students’ writing. Learning and Instruction, 91, Article 101894. https://doi.org/10.1016/j.learninstruc.2024.101894
47. Uyar, A., & Büyükahıska, D. (2025). Artificial intelligence as an automated essay scoring tool: A focus on ChatGPT. International Journal of Assessment Tools in Education, 12(1), 20 32. https://doi.org/10.21449/ijate.1517994
48. Wang, F., & Wang, S. (2012). A comparative study on the influence of automated evaluation system and teacher grading on students’ English writing. Procedia Engineering, 29, 993 997. https://doi.org/10.1016/j.proeng.2012.01.077
49. Wang, J., & Brown, M. (2007). Automated essay scoring versus human scoring: A comparative study. Journal of Technology, Learning, and Assessment, 6(2). https://ejournals.bc.edu/index.php/jtla/article/view/1632/1476
50. Wang, J., Hu, X., Hou, W., Chen, H., Zheng, R., Wang, Y., Yang, L., Huang, H., Ye, W., Geng, X., Jiao, B., Zhang, Y., & Xie, X. (2023). On the robustness of ChatGPT: An adversarial and out-of-distribution perspective. arXiv preprintarXiv:2302.12095. https://doi.org/10.48550/arXiv.2302.12095
51. Wang, K., Harrington, M., & White, P. (2012). Detecting breakdowns in local coherence in the writing of Chinese English learners. Journal of Computer Assisted Learning, 28(4), 396 410. https://doi.org/10.1111/j.1365-2729.2011.00475.x
52. Warschauer, M., Tseng, W., Yim, S., Webster, T., Jacob, S., Du, Q., & Tate, T. (2023). The affordances and contradictions of AI-generated text for writers of English as a second or foreign language. Journal of Second Language Writing, 62, Article 101071. https://doi.org/10.1016/j.jslw.2023.101071
53. Wu, H., Wang, W., Wan, Y., Jiao, W., & Lyu, M. (2023). ChatGPT or Grammarly? Evaluating ChatGPT on grammatical error correction benchmark. arXiv preprint arXiv:2303.13648. https://doi.org/10.48550/arXiv.2303.13648
54. Xia, W., Mao, S., & Zheng, C. (2024). Empirical study of large language models as automated essay scoring tools in English composition-taking TOEFL independent writing task for example. arXiv preprint arXiv:2401.03401. https://doi.org/10.48550/arXiv.2401.03401
55. Xiao, C., Ma, W., Song, Q., Xu, S.X., Zhang, K., Wang, Y., & Fu, Q. (2025). Human-ai collaborative essay scoring: A dual-process framework with LLMs. In Proceedings of the 15th International Learning Analytics and Knowledge Conference (pp. 293-305). Association for Computing Machinery. https://doi.org/10.1145/3706468.3706507
56. Yancey, K.P., Laflair, G., Verardi, A., & Burstein, J. (2023). Rating short L2 essays on the CEFR scale with GPT-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (pp. 576 584). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.bea-1.49
57. Yiğiter, M., & Boduroğlu, E. (2025). Examining the performance of artificial intelligence in scoring students' handwritten responses to open ended Items. Education and Science, 50, 1 18. https://doi.org/10.15390/EB.2025.14119
58. Yoon, S.Y., Miszoglad, E., & Pierce, L.R. (2023). Evaluation of ChatGPT feedback on ELL writers' coherence and cohesion. arXiv preprint arXiv:2310.06505. https://doi.org/10.48550/arXiv.2310.06505