Comparative Evaluation of Chatgpt, Gemini and Grok with and without Deep Research Mode in Answering Bone Augmentation Queries

Abstract

This study aimed to evaluate the information generation performance of four large language models (ChatGPT-o1-pro, ChatGPT-4o, Gemini 2.5 Flash, and Grok-3) using multidimensional criteria, based on their responses to expert-level questions in the field of intraoral bone augmentation. In addition, the study investigated the effect of enabling Deep Search mode on model performance. A total of 20 expert-generated questions, developed with input from domain specialists, were submitted to each model under two different retrieval configurations, including Deep Search. The responses were assessed independently by two periodontology experts in terms of accuracy, content quality, readability, and reading level. The findings indicated that the ChatGPT-o1-pro model with Deep Search achieved the highest and most balanced performance across all evaluation criteria. Similarly, ChatGPT-4o with Deep Search demonstrated strong performance, particularly in accuracy and content quality, although its readability was comparatively lower. In contrast, Gemini 2.5 Flash and Grok-3 showed relatively weaker performance across all dimensions. Overall, the results revealed that activating Deep Search mode provided a statistically significant improvement in performance. These findings underscore the growing potential of large language models in clinically complex and information-intensive domains such as dentistry. The results also suggest that, beyond model selection, the operational mode—particularly the use of advanced retrieval configurations—plays a crucial role in determining the quality of generated information.

Keywords

• Large language models
• ChatGPT
• Gemini
• Grok
• Artificial intelligence
• Clinical decision support systems

References

1. Najeeb M, Islam S. Artificial intelligence in restorative dentistry: current trends and future prospects. BMC Oral Health. 2025; 25:592.
2. Lee SJ, Poon J, Jindarojanakul A, et al. Artificial intelligence in dentistry: exploring emerging applications and future prospects. J Dent. 2025; 155:105648.
3. Giannakopoulos K, Kavadella A, Salim AA, et al. Evaluation of the performance of generative AI large language models ChatGPT, Google Bard, and Microsoft Bing Chat in supporting evidence-based dentistry: comparative mixed methods study. J Med Internet Res. 2023;25: e51580.
4. Dermata A, Arhakis A, Makrygiannakis MA, et al. Evaluating the evidence-based potential of six large language models in paediatric dentistry: a comparative study on generative artificial intelligence. Eur Arch Paediatr Dent. 2025; 26:527–35.
5. Macrì M, D'Albis V, D'Albis G, et al. The role and applications of artificial intelligence in dental implant planning: a systematic review. Bioengineering. 2024; 11:778.
6. Mukherjee J, Sharma R, Dutta P, Bhunia B. Artificial intelligence in healthcare: a mastery. Biotechnol Genet Eng Rev. 2024; 40:1659–708.
7. Aftab W, Apostolou Z, Bouazoune K, Straub T. Optimizing biomedical information retrieval with a keyword frequency-driven prompt enhancement strategy. BMC Bioinformatics. 2024; 25:281.
8. Maslej MM, Donner K, Thakur A, et al. Deriving insights from open-ended learner feedback: an exploration of natural language processing approaches. J Contin Educ Health Prof. 2025.

9. Su H, Sun Y, Li R, et al. Large language models in medical diagnostics: scoping review with bibliometric analysis. J Med Internet Res. 2025;27: e72062.
10. Wang L, Li J, Zhuang B, et al. Accuracy of large language models when answering clinical research questions: systematic review and network meta-analysis. J Med Internet Res. 2025;27: e64486.
11. Künzle P, Paris S. Performance of large language artificial intelligence models on solving restorative dentistry and endodontics student assessments. Clin Oral Investig. 2024; 28:575.
12. Suárez A, Jiménez J, Llorente de Pedro M, et al. Assessing ChatGPT's potential as an auxiliary intelligent virtual assistant in oral surgery. Comput Struct Biotechnol J. 2023; 24:46–52.
13. Thirunavukarasu AJ, Hassan R, Mahmood S, et al. Trialling a large language model in general practice with the applied knowledge test: observational study. JMIR Med Educ. 2023;9:e46599.
14. Freyer O, Wiest IC, Kather JN, Gilbert S. A future role for health applications of large language models depends on regulators enforcing safety standards. Lancet Digit Health. 2024;6: e662–72.
15. Matheny ME, Yang J, Smith JC, et al. Enhancing postmarketing surveillance of medical products with large language models. JAMA Netw Open. 2024;7: e2428276.
16. Goh E, Gallo R, Hom J, et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Netw Open. 2024;7: e2440969.
17. Gan W, Ouyang J, Li H, et al. Integrating ChatGPT in orthopedic education for medical undergraduates: randomized controlled trial. J Med Internet Res. 2024;26: e57037.
18. OpenAI. Introducing ChatGPT Pro. OpenAI; 2024. Available from: https://openai.com/index/introducing-chatgpt-pro/ (accessed 5 Dec 2024).
19. Balel Y, Zogo A, Yıldız S, Tanyeri H. Can ChatGPT-4o provide new systematic review ideas to oral and maxillofacial surgeons? J Stomatol Oral Maxillofac Surg. 2024; 125:101979.
20. Sau S, George DD, Singh R, et al. Accuracy and quality of ChatGPT-4o and Google Gemini performance on image-based neurosurgery board questions. Neurosurg Rev. 2025; 48:320.
21. Durmazpinar PM, Ekmekci E. Comparing diagnostic skills in endodontic cases: dental students versus ChatGPT-4o. BMC Oral Health. 2025; 25:457.
22. xAI. Grok: AI for understanding and generating human language. xAI; 2024. Available from: https://x.ai/grok (accessed Jul 2025).
23. Sozer A, Sahin MC, Sozer B, et al. Do LLMs have “the eye” for MRI? Evaluating GPT-4o, Grok, and Gemini on brain MRI performance. Diagnostics. 2025; 15:1320.
24. Büyüközer Özkan H, Doğan Çankaya T, Kölüş T. The impact of language variability on artificial intelligence performance in regenerative endodontics. Healthcare. 2025; 13:1190.
25. DeepMind. Introducing Gemini: our largest and most capable AI model. Google Blog; 2023. Available from: https://blog.google/technology/ai/google-gemini-ai/ (accessed Jul 2025).
26. Google. Grounding with Google Search (Gemini API documentation). Google; 2024. Available from: https://ai.google.dev/gemini-api/docs/google-search (accessed Jul 2025).
27. Solovyev I, Basu Mallick S. Gemini 2.0: level up your apps with real-time multimodal interactions. Google Developers Blog; 2024. Available from: https://developers.googleblog.com/2024/04/gemini-2-real-time-multimodal-interactions.html (accessed Jul 2025).
28. Buser D, Urban I, Monje A, et al. Guided bone regeneration in implant dentistry: basic principle, progress over 35 years, and recent research activities. Periodontol 2000. 2023; 93:9–25.
29. Urban IA, Montero E, Amerio E, et al. Techniques on vertical ridge augmentation: indications and effectiveness. Periodontol 2000. 2023; 93:153–82.
30. Alavi SE, Gholami M, Shahmabadi HE, et al. Resorbable GBR scaffolds in oral and maxillofacial tissue engineering: design, fabrication, and applications. J Clin Med. 2023; 12:6962.
31. Munakata M, Kataoka Y, Yamaguchi K, et al. Risk factors for early implant failure and selection of bone grafting materials for various bone augmentation procedures: a narrative review. Bioengineering. 2024; 11:192.
32. Kloss FR, Kämmerer PW, Kloss Brandstätter A. Risk factors for complications following staged alveolar ridge augmentation and dental implantation: a retrospective evaluation of 151 cases. J Clin Med. 2023; 12:6.
33. Urban I, Montero E, Sanz Sánchez I, et al. Minimal invasiveness in vertical ridge augmentation. Periodontol 2000. 2023; 91:126–44.
34. Dursun D, Bilici Geçer R. Can artificial intelligence models serve as patient information consultants in orthodontics? BMC Med Inform Decis Mak. 2024; 24:211.
35. Elkarmi R, Abu Ghazaleh S, Sonbol H, et al. ChatGPT for parents’ education about early childhood caries: a friend or foe? Int J Paediatr Dent. 2025; 35:717–24.
36. Flesch R. A new readability yardstick. J Appl Psychol. 1948; 32:221–33.
37. Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of new readability formulas for Navy enlisted personnel. Research Branch Report 8-75. Naval Technical Training Command; Millington, TN, USA; 1975.
38. Duarte ND, Frigério PB, Chica GEA, et al. Biomaterials for guided tissue regeneration and guided bone regeneration: a review. Dent J. 2025; 13:179.
39. Eldeeb AE, Salah S, Elkasabgy NA. Biomaterials for tissue engineering applications and current updates in the field: a comprehensive review. AAPS PharmSciTech. 2022; 23:267.
40. Inchingolo F, Inchingolo AM, Latini G, et al. Guided bone regeneration: CGF and PRF combined with various types of scaffolds—a systematic review. Int J Dent. 2024; 2024:4990295.
41. Farsi D. Social media and health care, part I: literature review of social media use by health care providers. J Med Internet Res. 2021;23: e23205.
42. Liu C, Staples R, Gómez-Cerezo MN, et al. Emerging technologies of three-dimensional printing and mobile health in COVID-19 immunity and regenerative dentistry. Tissue Eng Part C Methods. 2023; 29:163–82.
43. Preiksaitis C, Ashenburg N, Bunney G, et al. The role of large language models in transforming emergency medicine: scoping review. JMIR Med Inform. 2024;12: e53787.
44. Meng X, Yan X, Zhang K, et al. The application of large language models in medicine: a scoping review. iScience. 2024; 27:109713.
45. Li J, Guan Z, Wang J, et al. Integrated image-based deep learning and language models for primary diabetes care. Nat Med. 2024; 30:2886–96.
46. Goh E, Gallo R, Hom J, et al. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Netw Open. 2024;7: e2440969.
47. Cicek O, Yıldırım A. Evaluating the reliability of artificial intelligence in clinical decisions on early maxillary expansion: a comparative study of eight large language models. BMC Oral Health. 2026.
48. Stephan D, Bertsch A, Burwinkel M, et al. AI in dental radiology: improving the efficiency of reporting with ChatGPT. J Med Internet Res. 2024;26: e60684.
49. Brondani M, Alves C, Ribeiro C, et al. Artificial intelligence, ChatGPT, and dental education: implications for reflective assignments and qualitative research. J Dent Educ. 2024;88:1671–80.
50. Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2: e0000198.
51. Uehara O, Morikawa T, Harada F, et al. Performance of ChatGPT-3.5 and ChatGPT-4o in the Japanese national dental examination. J Dent Educ. 2024; 89:459–66.
52. Jin HK, Lee HE, Kim E. Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Med Educ. 2024; 24:1013.
53. Babaee Hemmati Y, Rasouli M, Falahchai M. Comparative analysis of ChatGPT-3.5 and GPT-4 in open-ended clinical reasoning across dental specialties. Eur J Dent Educ. 2025; Epub ahead of print.
54. Dashti M, Ghasemi S, Ghadimi N, et al. Performance of ChatGPT 3.5 and 4 on U.S. dental examinations: the INBDE, ADAT, and DAT. Imaging Sci Dent. 2024; 54:271–5.
55. De Vito A, Geremia N, Bavaro DF, et al. Comparing large language models for antibiotic prescribing in different clinical scenarios: which performs better? Clin Microbiol Infect. 2025; Epub ahead of print.
56. Ekmekci E, Durmazpinar PM. Evaluation of different artificial intelligence applications in responding to regenerative endodontic procedures. BMC Oral Health. 2025; 25:53.
57. Ahmed WM, Azhari AA, Alfaraj A, et al. The quality of AI-generated dental caries multiple choice questions: a comparative analysis of ChatGPT and Google Bard language models. Heliyon. 2024;10: e28198.
58. Kinikoglu I. Evaluating ChatGPT and Google Gemini performance and implications in Turkish dental education. Cureus. 2025;17: e77292.
59. Yetkin NA, Baran B, Rabahoğlu B, et al. Evaluating the reliability and quality of sarcoidosis-related information provided by AI chatbots. Healthcare. 2025; 13:1344.
60. Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare. 2023; 11:887.