A comparative evaluation of Artificial Intelligence Chatbots’ responses in special needs dentistry

Aims: This pilot study aimed to evaluate the accuracy and consistency of widely used Artificial Intelligence (AI) chatbots in responding to questions and diagnosing syndromes related to special needs dentistry. Methods: Three AI chatbots (ScholarGPT, ChatGPT, and Gemini) were assessed using ten true/false questions and ten diagnostic case-based questions derived from scientific literature. Each chatbot was queried three times over a three-week period. Responses were independently evaluated by two pediatric dentistry residents. Accuracy was scored dichotomously, while inter-reviewer reliability was measured using intraclass correlation coefficients (ICC). Statistical comparisons were conducted using the Wilcoxon exact test, and internal consistency was assessed with Cronbach’s alpha. Results: All chatbots demonstrated high inter-reviewer agreement (ICC >0.85, p<0.001). ScholarGPT achieved the highest diagnostic (57±6.3) and true/false (78±4.3) scores, followed by ChatGPT and Gemini. Differences between ScholarGPT and ChatGPT were not statistically significant; however, both significantly outperformed Gemini in most comparisons. Internal consistency was excellent for ScholarGPT (α=0.903), good for ChatGPT (α=0.887), and lower for Gemini (α=0.69). Overall accuracy remained moderate, particularly for diagnostic tasks. Conclusion: Although AI chatbots demonstrate consistent performance, their diagnostic accuracy in special needs dentistry remains insufficient for clinical application. These systems may serve as supportive educational tools; however, clinical decision-making should continue to rely on professional expertise until further advancements and validations are achieved.