tjph

Turkish Journal of Public Health

1304-1088

Turkish Society of Public Health Specialists

Health Services and Systems (Other)

Sağlık Hizmetleri ve Sistemleri (Diğer)

Assessing ChatGPT’s accuracy and reliability in medical education: a cross-sectional study

https://orcid.org/0009-0001-0497-529X

Vishal

A Ra

mvj medical college and research hospital

https://orcid.org/0009-0005-6896-1478

Harshitha

A S

mvj medical College and research hospital

https://orcid.org/0009-0005-4477-415X

Sindhu

A V

mvj medical College and research hospital

https://orcid.org/0009-0004-9689-5554

Abhivanth

mvj medical College and research hospital

https://orcid.org/0000-0002-5382-6653

Pavithra

MVJ Medical college and research institute

https://orcid.org/0000-0003-3814-5224

Madhukumar

Suwarna

mvj medical college and research hospital

04 25 2025

23 1 11 17 09 02 2024 03 29 2025

2003

Turkish Journal of Public Health

Objective: Artificial intelligence (AI), specifically ChatGPT, developed by Open AI provides human-like understanding and answers to a variety of domain questions and has the potential to transform medical education. However, its reliability in providing accurate clinical information is highly uncertain. This study is aimed at evaluating the accuracy and reliability of ChatGPT in answering multiple-choice questions (MCQs) and protocol-based questions in the field of medicine.Methods: This cross-sectional study was conducted using mixed methods at MVJ Medical College and Research Hospital (April 2024), Hoskote, India, i.e. MCQs (n=228) and protocol-based questions (n=10) from all 19 MBBS Subjects from standard medical literature were used to test ChatGPT. Subject experts checked the responses for accuracy. Statistical analysis, by chi-square test, was performed using IBM SPSS Version 20.0 for Windows.Results: The study findings stated that ChatGPT in easy and simple MCQs, had good accuracy, but its performance lowered with more complex questions, and overall answered about 57.02% of MCQs correctly. Protocol-based questions were given average scores, i.e. 6.35/10 for textbook accurate knowledge and 5.75/10 for real-life application.Conclusion: ChatGPT shows potential as a tool for medical education, especially in recalling basic facts but, it should not be relied upon as a sole source of information, instead used in conjunction with traditional methods to ensure a comprehensive understanding of medical concepts.

Application ChatGPT Knowledge MCQS Reliability

McCarthy, J., Minsky, M.L., Rochester, N. and Shannon, C.E. 2006. A Proposal for the Dartmouth Summer Re-search Project on Artificial Intelligence, August 31, 1955. AI Magazine. 27, 4 (Dec. 2006), 12. DOI:https://doi.org/10.1609/aimag.v27i4.1904.

Chen J. Playing to our human strengths to prepare medical students for the future. Korean J Med Educ. 2017;29(3):193-197. doi:10.3946/kjme.2017.65

Meskó B, Hetényi G, Győrffy Z. Will artificial intelligence solve the human resource crisis in healthcare? BMC Health Serv Res. 2018;18(1):545. Published 2018 Jul 13. doi:10.1186/s12913-018-3359-4

OpenAI. ChatGPT [Internet]. OpenAI API; 2022

Savery M, Abacha AB, Gayen S, Demner-Fushman D. Question-driven summarization of answers to consumer health questions. Sci Data. 2020;7(1):322. Published 2020 Oct 2. doi:10.1038/s41597-020-00667-z

Gutiérrez BJ, McNeal N, Washington C, Chen Y, Li L, Sun H, et al. Thinking about GPT-3 in-context learning for biomedical IE? Think again. arXiv. Preprint posted online on November 5, 2022. [doi: 10.48550/arXiv.2203.08410]

Kolachalama, V. B., & Garg, P. S. (2018). Machine learning and medical education. NPJ digital medicine, 1(1), 54.

Zarei M, Mamaghani HE, Abbasi A, Hosseini M. Application of artificial intelligence in medical education: A review of benefits, challenges, and solutions. Medicina Clínica Práctica. doi:10.1016/j.mcpsp.2023.100422

Sun L, Yin C, Xu Q, Zhao W. Artificial intelligence for healthcare and medical education: a systematic re-view. Am J Transl Res. 2023;15(7):4820-4828. Published 2023 Jul 15

Jin, Di & Pan, Eileen & Oufattole, Nassim & Weng, Wei-Hung & Fang, Hanyi & Szolovits, Peter. (2021). What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Ex-ams. Applied Sciences. 11. 6421. 10.3390/app11146421.

Ha LA, Yaneva V. Automatic question answering for medical MCQs: can it go further than information retriev-al? In: Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019). 2019 Presented at RANLP 2019; September 2-4, 2019; Varna, Bulgaria p. 418-422. [doi: 10.26615/978-954-452-056-4_049

Xu X, Chen Y, Miao J. Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review. J Educ Eval Health Prof [Internet]. 2024;21:6. [doi: 10.3352/jeehp.2024.21.6]

Mackey BP, Garabet R, Maule L, Tadesse A, Cross J, Weingarten M. Evaluating ChatGPT-4 in medical educa-tion: an assessment of subject exam performance reveals limitations in clinical curriculum support for stu-dents. Discov Artif Intell [Internet]. 2024;4(1). [doi: 10.1007/s44163-024-00135-2]

Harrison TR, Braunwald E. Harrison’s principles of internal medicine. 15th ed. New York, NY: McGraw-Hill; 2002

O’Connell PR, McCaskie AW, Sayers RD, editors. Bailey & love’s short practice of surgery - 28th edition. 28th ed. London, England: CRC Press; 2023.

Espinosa L, Salathé M. Use of large language models as a scalable approach to understanding public health discourse. PLOS Digit Health [Internet]. 2024;3(10):e0000631 [doi: 10.1371/journal.pdig.0000631]