Assessing the Performance of ChatGPT on Dentistry Specialization Exam Questions: A Comparative Study with DUS Examinees

Yıl 2025, Cilt: 7 Sayı: 1, 162 - 166, 15.01.2025

Mustafa Temiz , Ceylan Güzel

https://doi.org/10.37990/medr.1567242

Öz

Aim: This study aims to evaluate the performance of the ChatGPT-4.0 model in answering questions from the Turkish Dentistry Specialization Exam (DUS), comparing it with the performance of DUS examinees and exploring the model’s clinical reasoning capabilities and its potential educational value in dental training. The objective is to identify the strengths and limitations of ChatGPT when tasked with responding to questions typically presented in this critical examination for dental professionals.
Material and Method: The study analyzed DUS questions from the years 2012 to 2017, focusing on the basic medical sciences and clinical sciences sections. ChatGPT's responses to these questions were compared with the average scores of DUS examinees, who had previously taken the exam. A statistical analysis was performed to assess the significance of the differences in performance between ChatGPT and the human examinees.
Results: ChatGPT significantly outperformed DUS examinees in both the basic medical sciences and clinical sciences sections across all years analyzed. The statistical analysis revealed that the differences in performance between ChatGPT and DUS examinees were statistically significant, with ChatGPT demonstrating superior accuracy in all years.
Conclusion: ChatGPT’s performance on the DUS demonstrates its potential as a supplementary tool for dental education and exam preparation. However, future research should focus on integrating AI into practical dental training, particularly in assessing its real-world applicability. The limitations of AI in replicating hands-on clinical decision-making in unpredictable environments must also be considered.

Anahtar Kelimeler

Artificial intelligence in dentistry, clinical decision support, chatgpt in medical education, dental exam performance

Kaynakça

• T.C. Cumhurbaşkanlığı Mevzuat Bilgi Sistemi. Tıpta ve diş hekimliğinde uzmanlık eğitimi yönetmeliği. www.mevzuat.gov.tr/mevzuat?MevzuatNo=39700&Mevzuat Tur=7&MevzuatTertip=5 acces date 10.10.2024.
• Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. ArXiv. 2020 doi: 10.48550/arXiv.2005.14165
• Patino GA, Amiel JM, Brown M, et al. The promise and perils of artificial intelligence in health professions education practice and scholarship. Acad Med. 2024;99:477-81.
• Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2:e0000198.
• Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. J Med Syst. 2023;47:33.
• Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. ArXiv. May 2017. doi: 10.48550/arXiv.1706.03762.
• Nagi F, Salih R, Alzubaidi M. et al. Applications of artificial intelligence (AI) in medical education: a scoping review. Stud Health Technol Inform. 2023;305:648-51.
• Rajpurkar P, Irvin J, Ball RL, et al. Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med. 2018;15:e1002686.
• Cowan N. Working memory underpins cognitive development, learning, and education. Educ Psychol Rev. 2014;26:197-223.
• Sumbal A, Sumbal R, Amir A. Can ChatGPT-3.5 pass a medical exam? a systematic review of ChatGPT's performance in academic testing. J Med Educ Curric Dev. 2024;11:23821205241238641.
• Yu P, Fang C, Liu X, et al. Performance of ChatGPT on the Chinese postgraduate examination for clinical medicine: survey study. JMIR Med Educ. 2024;10:e48514.
• Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J. 2019;6:94-8.
• Choi W. Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQs. BMC Med Educ. 2023;23:864.
• Totlis T, Natsis K, Filos D, et al. The potential role of ChatGPT and artificial intelligence in anatomy education: a conversation with ChatGPT. Surg Radiol Anat. 2023;45:1321-9.
• Meo SA, Al-Masri AA, Alotaibi M, et al. ChatGPT knowledge evaluation in basic and clinical medical sciences: multiple choice question examination-based performance. Healthcare (Basel). 2023;11:2046.
• Clement J, Maldonado AQ. Augmenting the transplant team with artificial intelligence: toward meaningful AI use in solid organ transplant. Front Immunol. 2021;12:694222.
• Ouanes K, Farhah N. Effectiveness of artificial intelligence (AI) in clinical decision support systems and care delivery. J Med Syst. 2024;48:74.
• Pashkov VM, Harkusha AO, Harkusha YO. Artificial intelligence in medical practice: regulative issues and perspectıives. Wiad Lek. 2020;73:2722-7.
• Kelly CJ, Karthikesalingam A, Suleyman M, et al. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 2019;17:195.
• Benzinger L, Ursin F, Balke WT, et al. Should artificial intelligence be used to support clinical ethical decision-making? A systematic review of reasons. BMC Med Ethics. 2023;24:48.
• Mörch CM, Atsu S, Cai W. et al. Artificial intelligence and ethics in dentistry: a scoping review. J Dent Res. 2021;100:1452-60.
• Chen YW, Stanley K, Att W. Artificial intelligence in dentistry: current applications and future perspectives. Quintessence Int. 2020;51:248-57. Erratum in: Quintessence Int. 2020;51:430.
• Duggal I, Tripathi T. Ethical principles in dental healthcare: Relevance in the current technological era of artificial intelligence. J Oral Biol Craniofac Res. 2024;14:317-21.
• Sahin MC, Sozer A, Kuzucu P. et al. Beyond human in neurosurgical exams: ChatGPT's success in the Turkish neurosurgical society proficiency board exams. Comput Biol Med. 2024;169:10780