Üniversite Öğrencilerinin Yazılı Üretimlerinin OpenAI GPT ile Değerlendirilmesi

Year 2024, Volume: 14 Issue: 3, 121 - 134, 31.12.2024

Ayfer Sayın , Deniz Melanlıoğlu

https://doi.org/10.53478/yuksekogretim.1418870

Abstract

Yazılı üretim ortaya koyma öğrenciler tarafından her zaman zor bir görev olarak nitelenmektedir. Yazmaya ilişkin tutum, motivasyon, değerlendirme ölçütleri, konuya hâkimiyet, dili kullanma yetisi gibi değişkenler yazmaya önyargıyla yaklaşmaya neden olmaktadır. Ancak yazma sadece bir akademik başarı göstergesi değil aynı zamanda iş dünyasında da ihtiyaç duyulan bir beceridir. Bu nedenle üniversite eğitimi sırasında öğrencilerin etkili yazma becerisi kazanmaları önemli görülmektedir. Öğrencilerin yazmada yetkinlik kazanması, yazma uygulaması yapmaya ve ürünlere geribildirimde bulunmayla yakından ilişkilidir. Anlamlı dönüt verilmesinde yazılı üretimlerin objektif değerlendirilmesi gerekmektedir. Yazmaya yönelik objektif değerlendirmede bulunma bazen puanlayıcıların yaklaşımından kaynaklı olarak geçerli ve güvenilir sonuçlar vermeyebilir. Böyle bir durumda teknolojinin sunduğu imkânlardan faydalanılabileceğine inanılmaktadır. Bu bağlamda araştırmada, üniversite öğrencilerinin yazılı üretimlerinin insan puanlayıcılar ve yapay zekâ tarafından puanlanmıştır. Bu iki puanlamanın karşılaştırılarak incelenmesi araştırmanın amacını oluşturmuştur. Böylece yazılı üretimleri değerlendirmede OpenAI tarafından geliştirilen GPT yapay zekâ sistemlerinin kullanılabilirliği sınanacaktır. Söz konusu amaç doğrultusunda araştırma, ilişkisel tarama modelinde yürütülmüştür. Katılımcılar, gönüllülük esasına bağlı olarak bir devlet üniversitesinde öğrenim gören 60 birinci sınıf öğrencisidir. Araştırma kapsamında katılımcılardan yazılı üretim görevi doğrultusunda bir metin oluşturmaları istenmiş ve bunlar bütüncül puanlama anahtarı ile puanlanmıştır. Elde edilen verilerin analizi sonrasında GPT ile uzmanların puanları arasında pozitif yönde ve orta düzeyde bir ilişki olduğu saptanmıştır.

Keywords

OpenAI GPT, Türkçe Eğitimi, Yazma Becerisi, Bütüncül Puanlama, Üniversite Öğrencisi.

Assessment of University Students’ Essays through OpenAI GPT

Abstract

Students always characterize written essays as a difficult task. Variables such as attitude towards writing, motivation, evaluation criteria, mastery of the subject, and ability to use language cause prejudice towards writing. However, writing is an indicator of academic success and a skill needed in the business world. For this reason, it is considered essential for students to gain effective patching skills during university education. Students gaining competence in writing is closely related to writing practice and giving feedback to products. In providing meaningful feedback, written essays should be evaluated objectively. Objective evaluation of writing may sometimes not give valid and reliable results due to the approach of the scorers. In such a case, the opportunities offered by technology can be utilized. In this context, the study aims to examine the scoring of university students’ written essays by human raters and artificial intelligence, compare and analyze the two scorings, and test the usability of GPT artificial intelligence systems developed by OpenAI in evaluating written essays. The research was conducted in the relational survey model in line with this purpose. The participants were 60 first-year students studying at a state university voluntarily. Within the scope of the research, the participants were asked to create a text in line with the written essay task, and these were scored with a holistic scoring key. After the analysis of the data obtained, it was found that there was a positive and moderate relationship between GPT and experts’ scores.

Keywords

OpenAI GPT, Turkish Education, Writing Skill, Holistic Scoring, Higher Education Attendee

References

• Adams, D., & Chuah, K. M. (2022). Artificial intelligence-based tools in research writing: Current trends and future potentials. Artificial Intelligence in Higher Education, 169-184.
• Ashton, R. (2007). The write skills: Rob Ashton looks at the challenges of improving graduates’ business writing skills. Training Journal, 33-38.
• Athaluri, S., Manthena, S., Kesapragada, V., Yarlagadda, V., Dave, T., & Duddumpudi, R. (2023). Exploring the boundaries of reality: Investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references. Cureus.
• Attali, Y. (2016). A comparison of newly-trained and experienced raters on a standardized writing assessment. Language Testing, 33(1), 99–115.
• Aydın, Ö., & Karaarslan, E. (2022). PENAI ChatGPT generated literature review: Digital twin in healthcare. In Ö. Aydın (Ed.), Emerging Computer Technologies 2 (pp. 22-31). İzmir Akademi Derneği.
• Bacon, D. R., & Anderson, E. S. (2004). Assessing and enhancing the basic writing skills of marketing students. Business Communication Quarterly, 67(4), 443-455.
• Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. SSRN. https://doi.org/10.2139/ssrn.4337484
• Banachewicz, K., Massaron, L., & Goldbloom, A. (2022). The Kaggle Book: Data Analysis And Machine Learning For Competitive Data Science. Birmingham: Packt Publishing Ltd.
• Barker, R. T., & Hall, B. S. (1995). Using the business briefing to develop oral communication skills. Journal of Management Education, 19(4), 513-518.
• Benjamin, R., & Chun, M. (2003). A new field of dreams: The collegiate learning assessment project. Peer Review, 5(4), 26-29.
• Bilgen, Ö. B., & Doğan, N. (2017). Puanlayıcılar arası güvenirlik belirleme tekniklerinin karşılaştırılması. Journal of Measurement and Evaluation in Education and Psychology, 8(1), 63-78.
• Bishop, L. (2023). A computer wrote this paper: What ChatGPT means for education, research, and writing. Research, and Writing, 26.
• Brandt, D. (2005). Writing for a living: Literacy and the knowledge economy. Written Communication, 22(2), 166-197.
• Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., & Askell, A. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901.
• Büyüköztürk, Ş. (2018). Sosyal bilimler için veri analizi el kitabı. Pegem Atıf İndeksi, 001-214.
• Charney, D. (1984). The validity of using holistic scoring to evaluating writing: A critical overview. Research in the Teaching of English, 18, 65-81.
• Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., & Brockman, G. (2021). Evaluating large language models trained on code. arXiv preprint, arXiv:2107.03374.
• Chenoweth, N. A., & Hayes, J. R. (2001). Fluency in writing: Generating text in L1 and L2. Written Communication, 18(1), 80–98.
• Chodorow, M., & Burstein, J. (2004). Beyond essay length: Evaluating e-rater’s performance on TOEFL essays. ETS Research Reports, i–38.
• Coşkun, E., & Tiryaki, E. N. (2011). Tartışmacı metin yapısı ve öğretimi. Mustafa Kemal Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, 8(16), 63-73.
• Crossley, S. (2020). Linguistic features in writing quality and development: An overview. Journal of Writing Research, 11(3), 415–443.
• Crosthwaite, P., Storch, N., & Schweinberger, M. (2020). Less is more? The impact of written corrective feedback on corpus-assisted L2 error resolution. Journal of Second Language Writing, 49.
• Deane, P. (2013). On the relation between automated essay scoring and modern views of the writing construct. Assessing Writing, 18(1), 7–24.
• Enos, M. F. (2010). Instructional interventions for improving proofreading and editing skills of college students. Business Communication Quarterly, 73(3), 265-281.
• Fisher, A. (1999). Ask Annie. Fortune, 145(5), 223-225.
• Fleiss, J. L., Nee, J. C., & Landis, J. R. (1979). Large sample variance of kappa in the case of different sets of raters. Psychological Bulletin, 86(5), 974.
• Floridi, L., & Chiriatti, M. (2020). GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30, 681-694.
• Göçer, A. (2010). Eğitim fakültesi öğrencilerinin yazılı anlatım becerilerinin süreç yaklaşımı ve metinsellik ölçütleri ekseninde değerlendirilmesi: Niğde Üniversitesi örneği. Kastamonu Eğitim Dergisi, 18(1), 271-290.
• Graham, S., Harris, K. R., & Hebert, M. (2011). It is more than just the message: Presentation effects in scoring writing. Focus Exceptional Children, 44(4), 1–12.
• Greenberg, K. L. (1988). Effective writing choices and conventions. St. Martin’s Press.
• Guo, L., Crossley, S. A., & McNamara, D. S. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18(3), 218–238.
• Günay, D. (2007). Metin bilgisi. Multilingual Yayınları.
• Henricks, M. (2007). Writing skills are vital for today’s employees, but few have them. Entrepreneur, 35(7), 85-86.
• Hirschberg, J., & Manning, C. D. (2015). Advances in natural language processing. Science, 349(6245), 261-266.
• Hsu, H. C. (2019). Wiki-mediated collaboration and its association with L2 writing development: An exploratory study. Computer Assisted Language Learning, 32(8), 945–967.
• Huot, B. (1990). The literature of direct writing assessment: Major concerns and prevailing trends. Review of Educational Research, 60(2), 237-263.
• Imran, M., & Almusharraf, N. (2023). Analyzing the role of ChatGPT as a writing assistant at higher education level: A systematic review of the literature. Contemporary Educational Technology, 15(4).
• Ivanov, S., & Soliman, M. (2023). Game of algorithms: ChatGPT implications for the future of tourism education and research. Journal of Tourism Futures, 9(2), 214-221.
• Jabotinsky, H. Y., & Sarel, R. (2022). Co-authoring with an AI? Ethical dilemmas and artificial intelligence. Arizona State Law Journal. Available at SSRN: or
• Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W. (2023, April). ChatGPT and software testing education: Promises & perils. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 4130-4137). IEEE.
• Jiao, W., Wang, W., Huang, J. T., Wang, X., & Tu, Z. (2023). Is ChatGPT a good translator? A preliminary study. arXiv preprint, arXiv:2301.08745.
• Kahraman, A., & Yalvaç, F. (2015). EFL Turkish university students’ preferences about teacher feedback and its importance. Procedia - Social and Behavioral Sciences, 199, 73–80.
• Kalı, G. (2016). Türkçe öğretmeni adaylarının öyküleyici anlatımlarının bağdaşıklık ve tutarlılık açısından incelenmesi [Yayımlanmamış yüksek lisans tezi]. Muğla Sıtkı Koçman Üniversitesi Eğitim Bilimleri Enstitüsü.
• Kamnis, S. (2023). Generative pre-trained transformers (GPT) for surface engineering. Surface and Coatings Technology, 129680.
• Kang, Y., Cai, Z., Tan, C.-W., Huang, Q., & Liu, H. (2020). Natural language processing (NLP) in management research: A literature review. Journal of Management Analytics, 7(2), 139–172.
• Kellogg, R. T., & Raulerson, B. A. (2007). Improving the writing skills of college students. Psychonomic Bulletin & Review, 14(2), 237–242.
• Kroll, B. (1998). Assessing writing abilities. Annual Review of Applied Linguistics, 18, 219–240.
• Kyparisis, J. (1987). Sensitivity analysis framework for variational inequalities. Mathematical Programming, 38, 203–213.
• Lee, L. (2020). An exploratory study of using personal blogs for L2 writing in fully online language courses. In B. Zou & M. Thomas (Eds.), Recent developments in technology-enhanced and computer-assisted language learning (pp. 145–163). Information Science Reference.
• Liu, L., & Gibson, D. (2023). Exploring the use of ChatGPT for learning and research: Content data analysis and concerns. Society for Information Technology & Teacher Education International Conference (March 13, 2023, New Orleans, LA, United States).
• Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). Lost in the middle: How language models use long contexts. arXiv preprint, arXiv:2307.03172.
• Lund, B. D., & Wang, T. (2023). Chatting about ChatGPT: How may AI and GPT impact academia and libraries? Library Hi Tech News.
• Marcoulides, G. A. (1998). Applied generalizability theory models. In G. A. Marcoulides (Ed.), Modern methods of business research (pp. 131–158). Lawrence Erlbaum Associates.
• May, G. L., Thompson, M. A., & Hebblethwaite, J. (2012). A process for assessing and improving business writing at the MBA level. Business Communication Quarterly, 75(3), 252–270.
• McNamara, D. S., Crossley, S. A., Roscoe, R. D., Allen, L. K., & Dai, J. (2015). A hierarchical classification approach to automated essay scoring. Assessing Writing, 23, 35–59.
• Mhlanga, D. (2023). The value of Open AI and ChatGPT for the current learning environments and the potential future uses. SSRN Electronic Journal.
• Michel-Villarreal, R., Vilalta-Perdomo, E., Salinas-Navarro, D. E., Thierry-Aguilera, R., & Gerardou, F. S. (2023). Challenges and opportunities of generative AI for higher education as explained by ChatGPT. Education Sciences, 13(9). https://doi.org/10.3390/educsci13090856
• Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 100050.
• Monis, M., & Rodriques, M. V. (2012). Teaching creative writing in English language classroom. Indian Streams Research Journal, 2(10), 1–7.
• NAEP (National Assessment of Educational Progress). (2002). The nation’s report card: Writing 2002 major results. Retrieved from
• Nagata, R., Hashiguchi, T., & Sadoun, D. (2020). Is the simplest chatbot effective in English writing learning assistance? In L.-M. Nguyen, X.-H. Phan, K. Hasida, & S. Tojo (Eds.), Computational linguistics (pp. 245–256). Springer.
• Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2.
• OpenAI. (2022, November 30). ChatGPT: Optimizing language models for dialogue. Retrieved from
• Pavlik, J. V. (2023). Collaborating with ChatGPT: Considering the implications of generative artificial intelligence for journalism and media education. Journalism & Mass Communication Educator, 78(1), 84–93. https://doi.org/10.1177/10776958221149577
• Powers, D. E. (2005). Wordiness: A selective review of its influence, and suggestions for investigating its relevance in tests requiring extended written responses. ETS Research Reports, i–14.
• Quible, Z. K., & Griffin, F. (2007). Are writing deficiencies creating a lost generation of business writers? Journal of Education for Business, 83(1), 32–36.
• Quinlan, T., Higgins, D., & Wolff, S. (2009). Evaluating the construct-coverage of the e-rater scoring engine. ETS Research Reports, i–35.
• Ramachandran, L., Cheng, J., & Foltz, P. (2015). Identifying patterns for short answer scoring using graph-based lexico-semantic text matching. Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, Colorado.
• Ramalingam, V., Pandian, A., Chetry, P., & Nigam, H. (2018, January). Automated essay grading using machine learning algorithm. Journal of Physics: Conference Series.
• Ramesh, D., & Sanampudi, S. K. (2022). An automated essay scoring systems: A systematic literature review. Artificial Intelligence Review, 55(3), 2495–2527.
• Riordan, D. A., Riordan, M. P., & Sullivan, M. C. (2000). Writing across the accounting curriculum: An experiment. Business Communication Quarterly, 63(3), 49–59.
• Rowh, M. (2006). Write well, go far: It’s the skill every employer demands. Here’s how to build it. Career World: A Weekly Reader Publication, 34(4), 18–23.
• Saravia, E. (2018). Deep learning for NLP: An overview of recent trends. Retrieved November, 27, 2018.
• Schoonen, R. (2005). Generalizability of writing scores: An application of structural equation modeling. Language Testing, 22(1), 1–30.
• Schoonen, R., Vergeer, M., & Eiting, M. (1997). The assessment of writing ability: Expert readers versus lay readers. Language Testing, 14(2), 157–184.
• Seçkin, P., Arslan, N., & Ergenç, S. (2014). Bağdaşıklık ve tutarlılık bakımından lise ve üniversite öğrencilerinin yazılı anlatım becerileri. Uluslararası Türkçe Edebiyat Kültür Eğitim Dergisi, 3(1), 340–353.
• Smerd, J. (2007). New workers solely lacking literacy skills. Workforce Management, 86(21), 6.
• Stangor, C., & Walinga, J. (2019). Psychologists use descriptive, correlational, and experimental research designs to understand behaviour. In Introduction to Psychology.
• Stevens, B. (2005). What communication skills do employers want? Silicon Valley recruiters respond. Journal of Employment Counseling, 42(1), 2–9.
• Stokel-Walker, C. (2022). AI bot ChatGPT writes smart essays—Should professors worry? Nature, 613, 620–621.
• Susnjak, T. (2022). ChatGPT: The end of online exam integrity? Retrieved from
• Tan, R. G. R., Aviso, K. B., & Uy, O. M. (2015). Comprehensive sensitivity analysis in NLP models in PSE applications using space-filling DOE strategy. Chemical Engineering Transactions, 45, 523–528.
• Talan, T., & Kalınkara, Y. (2023). The role of artificial intelligence in higher education: ChatGPT assessment for anatomy course. Uluslararası Yönetim Bilişim Sistemleri ve Bilgisayar Bilimleri Dergisi, 7(1).
• Taylor, R. (1990). Interpretation of the correlation coefficient: A basic review. Journal of Diagnostic Medical Sonography, 6(1), 35–39.
• Tuzi, F. (2004). The impact of e-feedback on the revisions of L2 writers in an academic writing course. Computers and Composition, 21(2), 217–235.
• Wenzlaff, K., & Spaeth, S. (2022). Smarter than humans? Validating how OpenAI’s ChatGPT model explains crowdfunding, alternative finance and community finance. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4302443
• Wolfe, E. W., Song, T., & Jiao, H. (2016). Features of difficult-to-score essays. Assessing Writing, 27, 1–10.
• Xu, H., & Lv, Y. (2022). Mining and application of tourism online review text based on natural language processing and text classification technology. Wireless Communications and Mobile Computing, 2022.
• Yang, R., Cao, J., Wen, Z., Wu, Y., & He, X. (2020). Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking. Findings of the Association for Computational Linguistics: EMNLP 2020, 1560–1569.
• Yeadon, W., Inyang, O.-O., Mizouri, A., Peach, A., & Testrow, C. (2022). The death of the short-form physics essay in the coming AI revolution. Physics Education, 58, 1–13.
• Zech, J. M., Steele, R., Foley, V. K., & Hull, T. D. (2022). Automatic rating of therapist facilitative interpersonal skills in text: A natural language processing application. Frontiers in Digital Health, 4, 917918.
• Zhang, Z., Han, X., Zhou, H., Ke, P., Gu, Y., Ye, D., Qin, Y., Su, Y., Ji, H., & Guan, J. (2021). CPM: A large-scale generative Chinese pre-trained language model. AI Open, 2, 93–99.
• Zhu, X., Zhu, J., Li, H., Wu, X., Li, H., Wang, X., & Dai, J. (2022). Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.