A Qualitative Evaluation of ChatGPT-Generated Academic Writing in Higher Education

Abstract

The purpose of this study is to evaluate academic texts generated by ChatGPT, in terms of academic language use, structural integrity, content quality, and source reliability. Within this scope, ChatGPT-4 was prompted to produce an academic text in Turkish on a specific topic, and the resulting text was assessed using a rubric developed by the researchers. Subsequently, the text was qualitatively analyzed through detailed document analysis. The findings reveal that while ChatGPT generally demonstrates adequate performance in academic language use (such as objective tone, academic style, and proper application of academic grammatical structures), certain shortcomings were observed in critical analysis and adherence to APA 7 formatting rules. Although the organizational structure of the text presents a largely coherent and logical flow, minor structural weaknesses (such as the lack of a roadmap in the introduction) were identified. In terms of content, arguments remained mostly superficial, the literature was not examined in depth, and some references could not be verified. Furthermore, instances of information being inconsistent with or misinterpreted from the cited sources were detected. These results indicate that generative AI tools can be supportive in academic writing, but their outputs require careful evaluation.

Keywords

• Generative artificial intelligence
• Academic writing
• ChatGPT
• Content evaluation
• academic writing ethics

Supporting Institution

The authors declare that they did not receive any financial support from any institution or organization for the work presented in their article.

Ethical Statement

The author acknowledges that their work is not subject to ethics committee approval and that the Committee on Publication has been consulted throughout the entire study process. It is stated that the rules set forth by 'Cope of Ethics ' (COPE) are complied with.

Thanks

I would like to express my sincere gratitude to Burak Tüfekçioğlu, Tuğba Demirtaş, and Bayram Arıcı for their invaluable contributions to the development of the academic writing evaluation rubric used in this study. Their expertise and thoughtful feedback were instrumental in shaping the framework of this research. I also extend my deep appreciation to Burçin Coşkun for her dedicated efforts in evaluating the sample text examined in this study. Her careful analysis and attention to detail significantly strengthened the quality of our findings.

References

1. Cho, K., & MacArthur, C. (2010). Student revision with peer and expert reviewing. Learning and Instruction, 20(4), 328–338. https://doi.org/10.1016/j.learninstruc.2009.08.006
2. Falchikov, N. (2005). Improving assessment through student involvement: Practical solutions for aiding learning in higher and further education. RoutledgeFalmer.
3. Gielen, S., Peeters, E., & Dochy, F. (2010). Improving the effectiveness of peer feedback for learning. Learning and Instruction, 20(4), 304–315.
4. Li, L., Xiong, Y., & Zang, X. (2020). Peer assessment in the digital age: A meta-analysis comparing peer and teacher ratings. Assessment & Evaluation in Higher Education, 45(1), 20–42. https://doi.org/10.1080/02602938.2019.1600185
5. Liu, N. F., & Carless, D. (2006). Peer feedback: The learning element of peer assessment. Teaching in Higher Education, 11(3), 279–290.
6. Lu, R., & Zhang, Z. (2012). Assessing student collaborative problem-solving processes using chat data in online environments. Computers & Education, 59(2), 416–427.
7. Nicol, D. J., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher education: A peer review perspective. Assessment & Evaluation in Higher Education, 39(1), 102–122.
8. Panadero, E., & Jonsson, A. (2013). The use of scoring rubrics for formative assessment purposes revisited: A review. Educational Research Review, 9, 129–144.

9. Panadero, E., & Romero, M. (2014). To rubric or not to rubric? The effects of self-assessment on self-regulation, performance and self-efficacy. Assessment in Education: Principles, Policy & Practice, 21(2), 133–148.
10. Topping, K. J. (2009). Peer assessment. Theory into Practice, 48(1), 20–27.
11. Topping, K. J. (2017). Peer assessment: Learning by judging and discussing the work of other learners. Interdisciplinary Education and Psychology, 1(1), 1–17.
12. van Zundert, M., Sluijsmans, D., & van Merriënboer, J. (2010). Effective peer assessment processes: Research findings and future directions. Learning and Instruction, 20(4), 270–279.
13. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.