Comparing the Performance of Medical Students, ChatGPT-3.5 and ChatGPT-4 in Biostatistics Exam: Pros and Cons as an Education Assistant.

Year 2023, , 85 - 94, 30.12.2023

Ömer Faruk Asker , Emrah Gökay Özgür , Alper Eriç , Nural Bekiroğlu

https://doi.org/10.33461/uybisbbd.1329650

Abstract

Studies have shown that the level of knowledge in biostatistics among medical students is lower than expected. This situation calls for the need to implement new methods in biostatistics education. The aim of this study is to evaluate the feasibility of ChatGPT as an education assistant in biostatistics. ChatGPT is a natural language processing model developed by OpenAI. It provides human-like responses to questions asked by users and is utilized in various fields for gaining information. ChatGPT operates with the latest GPT-4 model, while the previous version, GPT-3.5, is still in use. In this study the biostatistics performance of 245 Marmara University School of Medicine students was compared to ChatGPT-3.5 and ChatGPT-4 using an exam covering basic biostatistics topics. According to findings, ChatGPT-3.5 achieved 80% success rate in the exam, while ChatGPT-4 achieved 100% success rate. In contrast, the students achieved 67.9% success rate. Furthermore, ChatGPT-3.5 only recorded 33% success rate in questions requiring mathematical calculations, while ChatGPT-4 achieved 100% success rate in these questions. In conclusion, ChatGPT is a potential education assistant in biostatistics. Its success has increased significantly in the current version compared to the previous one. Further studies will be needed as new versions are released.

Keywords

ChatGPT, Biostatistics, Education, NLP

Tıp Öğrencilerinin Biyoistatistik Sınavında ChatGPT-3.5 ve ChatGPT-4 Performanslarının Karşılaştırılması: Bir Eğitim Asistanı Olarak Artıları ve Eksileri

Abstract

Araştırmalar, tıp öğrencilerinin biyoistatistik konusundaki bilgi düzeylerinin beklenenden düşük olduğunu göstermiştir. Bu durum biyoistatistik eğitiminde yeni yöntemlerin uygulanması ihtiyacını doğurmaktadır. Bu çalışmanın amacı, ChatGPT'nin biyoistatistik alanında bir eğitim asistanı olarak uygulanabilirliğini değerlendirmektir. ChatGPT, OpenAI tarafından geliştirilmiş bir doğal dil işleme modelidir. Kullanıcılar tarafından sorulan sorulara insan benzeri cevaplar vermekte ve bilgi edinmek için çeşitli alanlarda kullanılmaktadır. ChatGPT, en yeni GPT-4 modeliyle çalışırken, önceki sürüm olan GPT-3.5 halen kullanımdadır. Bu çalışmada da 245 Marmara Üniversitesi Tıp Fakültesi öğrencisinin biyoistatistik performansları, temel biyoistatistik konularını kapsayan bir sınav kullanılarak ChatGPT-3.5 ve ChatGPT-4 ile karşılaştırıldı. SonuçlarElde edilen bulgulara göre ChatGPT-3.5 sınavda %80, ChatGPT-4 ise %100 başarı oranı elde etmiştir. Buna karşılık, öğrenciler %67,9 başarı oranı elde ettiler. Ayrıca ChatGPT-3.5 matematiksel hesaplama gerektiren sorularda sadece %33 başarı oranı kaydederken, ChatGPT-4 bu sorularda %100 başarı oranı elde etmiştir. Sonuç olarak ChatGPT, biyoistatistik alanında potansiyel bir eğitim asistanıdır. Mevcut sürümdeki başarısı önceki sürüme göre önemli ölçüde artmıştır. Yeni sürümler çıktıkça daha fazla çalışmaya ihtiyaç duyulacaktır.

Keywords

ChatGPT, Biyoistatistik, Eğitim, DDİ

References

• Bhat YA, Saeed G, Sahel SG, Almesned A, Alqwaee A, Al-Akhfash A. 2022. Evaluation of Basic Statistical Knowledge Among Medical Residents Published Article. Cardiology & Vascular Research.
• Brearley AM, Rott KW, Le LJ. 2023. A Biostatistical Literacy Course: Teaching Medical and Public Health Professionals to Read and Interpret Statistics in the Published Literature. Journal of Statistics and Data Science Education.
• Celik Y. 2019. The Importance of Biostatistical Methods in the “Evidence-Based Medicine”. International Journal of Basic and Clinical Studies (IJBCS). 8(1):1-7.
• Chiang CL, Zelen M. 1985. What Is Biostatistics?. Biometrics. 41(3):771.
• Choi JH, Hickman KE, Monahan A, Schwarcz DB. ChatGPT Goes to Law School. 2023. Minnesota Legal Studies Research Paper No. 23-03. [accessed 2023 March 26]. http://dx.doi.org/10.2139/ssrn.4335905.
• Couture F, Nguyen DD, Bhojani N, Lee JY, Richard PO. 2020. Knowledge and confidence level of Canadian urology residents toward biostatistics: A national survey. Canadian Urological Association Journal. 14(10).
• Frieder S, Pinchetti L, Griffiths RR, Salvatori T, Lukasiewicz T, Petersen PC, Chevalier, A, Berner J. 2023. Mathematical Capabilities of ChatGPT (Version 1). arXiv:2301.13867 [accessed 2023 March 26]
• Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. 2023. How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Medical Education, 9:e45312.
• GPT-4 is OpenAI’s most advanced system, producing safer and more useful responses. 2023. California: OpenAI; [accessed 2023 March 26]. https://openai.com/product/gpt-4.
• GPT-4. 2023. California: OpenAI; [Accessed 2023 March 26]. https://openai.com/research/gpt-4.
• Gruzieva TS, Stuchynska NV, Inshakova HV. 2020. Research on the effectiveness of teaching biostatistics of future physicians. Wiadomości Lekarskie. 73(10):2227–2232.
• Hanif A, Ajmal T. 2011. Statistical Errors in Medical Journals (A Critical Appraisal). Annals. 17(2):178-182.
• Jeblick K, Schachtner B, Dexl J, Mittermeier A, Stüber AT, Topalis J, Weber T, Wesp P, Sabel B, Ricke J, Ingrisch M. 2022. ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports (Version 1). arXiv.2212.14882. [accessed 2023 March 26]
• KEYPS: Kurumsal Egitim Yonetim ve Planlama Sistemi. 2023. Ankara: KEYPS; [accessed 2023 March 26]. www.keyps.com.tr/.
• Khan RA, Jawaid M, Khan AR, Sajjad M. 2023. ChatGPT - Reshaping medical education and clinical management. Pakistan Journal of Medical Sciences, 39(2).
• Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, Madriaga M, Aggabao R, Diaz-Candido G, Maningo J, Tseng V. 2023. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2(2):e0000198.
• Kurian N, Cherian JM, Sudharson NA, Varghese KG, Wadhwa S. 2023. AI is now everywhere. British Dental Journal, 234(2): 72–72.
• Mbakwe AB, Lourentzou I, Celi LA, Mechanic OJ, Dagan A. 2023. ChatGPT passing USMLE shines a spotlight on the flaws of medical education. PLOS Digital Health. 2(2):e0000205.
• Msaouel P, Kappos T, Tasoulis A, Apostolopoulos AP, Lekkas I, Tripodaki ES, Keramaris NC. 2014. Assessment of cognitive biases and biostatistics knowledge of medical residents: a multicenter, cross-sectional questionnaire study. Medical Education Online. 19(1):23646.
• Singh JP, Neupane S, Mehta RK, Deo GP. 2022. Assessing undergraduate students’ knowledge regarding application of biostatistics in research at medical college. Journal of Chitwan Medical College. 12(2):3–5.
• Taecharungroj V. 2023. “What Can ChatGPT Do?” Analyzing Early Reactions to the Innovative AI Chatbot on Twitter. Big Data and Cognitive Computing, 7(1):35
• Talan, T. & Kalınkara, Y. (2023). The Role of Artificial Intelligence in Higher Education: ChatGPT Assessment for Anatomy Course. Uluslararası Yönetim Bilişim Sistemleri ve Bilgisayar Bilimleri Dergisi, 7(1), 33-40. DOI: 10.33461/uybisbbd.1244777
• Tomak L, Civanbay H. 2022. Evaluation of biostatistics knowledge and skills of medical faculty students. Journal of Experimental and Clinical Medicine. 19(3):620–627.
• Vera-Ponce VJ, Torres-Malca JR, La Cruz-Vargas JAD, Zuzunaga Montoya FE, Chavez P H, Talavera-Ramirez JE, Cruz-Ausejo L. 2022. Analysis of Statistical Knowledge of Peruvian Medical Students: A Cross-Sectional Analytical Study Based on a Survey. International Journal of Statistics in Medical Research. 11:59–65.
• Wang X, Gong Z, Wang G, Jia J, Xu Y, Zhao J, Fan Q, Wu S, Hu W, Li X. 2023. ChatGPT Performs on the Chinese National Medical Licensing Examination.