Comparative Evaluation of Large Language Models in Prosthodontic Complication Scenarios
Öz
Aim: This study aimed to evaluate and compare the performance of 4 LLMs in responding to common prosthodontic complication scenarios.
Methods: An expert panel design was employed to assess the quality of responses generated by the four LLMs across five clinically relevant prosthodontic scenarios. Each scenario was entered 15 times into each model, yielding 300 responses in total. 7 prosthodontists independently evaluated the anonymized responses based on four criteria: clinical accuracy, applicability, clarity, and comprehensiveness, using a 5-point Likert scale. Descriptive statistics, one-way ANOVA, and post-hoc pairwise comparisons were performed to detect inter-model differences. The internal consistency (reliability) of evaluators’ ratings for each model was assessed using Cronbach’s alpha.
Results: Significant differences were found among the models for all evaluation criteria (p<.05). ChatGPT achieved the highest mean scores for clinical accuracy (3.89), clarity (3.90), and comprehensiveness (4.09), while Gemini showed the highest applicability (3.88). Cronbach’s alpha coefficients indicated strong reliability across all domains (0.776–0.880), with ChatGPT showing the highest single-domain coefficient (α=0.820–0.880).
Conclusion: Among the evaluated models, ChatGPT and Gemini generally received higher ratings on specific criteria in pairwise comparisons; however, the four models should not be interpreted as forming a definitive overall ranking. While LLMs show potential as supportive tools in prosthodontic education and clinical decision-making, their outputs should not replace expert judgment.
Anahtar Kelimeler
Prosthodontics, artificial intelligence, clinical decision support systems, complications, dental prostheses
Etik Beyan
Ethical approval of the study was received from the Committee for Ethics Research at the University of Atatürk (Date: 25/02/2026, Number: 15).
Kaynakça
- 1. Deiana G, Dettori M, Arghittu A, Azara A, Gabutti G, Castiglia P. Artificial Intelligence and Public Health: Evaluating ChatGPT Responses to Vaccination Myths and Misconceptions. Vaccines. 2023 Jul 7;11(7):1217. doi: 10.3390/vaccines11071217.
- 2. Ayers JW, Poliak A, Dredze M, Leas EC, Zhu Z, Kelley JB, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern Med. 2023 Jun 1;183(6):589–596. doi: 10.1001/jamainternmed.2023.1838.
- 3. Muthukrishnan N, Maleki F, Ovens K, Reinhold C, Forghani B, Forghani R. Brief History of Artificial Intelligence. Neuroimaging Clin N Am. 2020 Nov;30(4):393–399.doi: 10.1016/j.nic.2020.07.004.
- 4. Behers BJ, Vargas IA, Behers BM, Rosario MA, Wojtas CN, Deevers AC, et al. Assessing the Readability of Patient Education Materials on Cardiac Catheterization From Artificial Intelligence Chatbots: An Observational Cross-Sectional Study. Cureus. 2024 Jul 4;16(7):e63865. doi:10.7759/cureus.63865.
- 5. ACAR AH. Can natural language processing serve as a consultant in oral surgery? J Stomatol Oral Maxillofac Surg. 2024 Jun;125(3):101724. doi: 10.1016/j.jormas.2023.101724.
- 6. Ali R, Tang OY, Connolly ID, Fridley JS, Shin JH, Zadnik Sullivan PL, et al. Performance of ChatGPT, GPT-4, and Google Bard on a Neurosurgery Oral Boards Preparation Question Bank. Neurosurgery. 2023 Nov 1;93(5):1090–1098. doi: 10.1227/ neu.0000000000002551.
- 7. Bernstein IA, Zhang Y, Govil D, Majid I, Chang RT, Sun Y, et al. Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions. JAMA Netw Open. 2023 Aug 1;6(8):e2330320. doi: 10.1001/ jamanetworkopen.2023.30320.
- 8. Liu J, Wang C, Liu S. Utility of ChatGPT in Clinical Practice. J Med Internet Res. 2023 Jun 28:25:e48568. doi: 10.2196/48568.
- 9. Pan A, Musheyev D, Bockelman D, Loeb S, Kabarriti AE. Assessment of Artificial Intelligence Chatbot Responses to Top Searched Queries about Cancer. JAMA Oncol. 2023 Oct 1;9(10):1437–1440. doi: 10.1001/jamaoncol.2023.2947.
- 10. Felton D, Cooper L, Duqum I, Minsley G, Guckes A, Haug S, et al. Evidence-Based Guidelines for the Care and Maintenance of Complete Dentures: A Publication of the American College of Prosthodontists. J Prosthodont. 2011 Feb:20 Suppl 1:S1-S12. doi: 10.1111/j.1532-849X.2010.00683.x.