EVALUATING THE SUCCESS OF ARTIFICIAL INTELLIGENCE CHATBOTS IN ANSWERING COMMON ORTHODONTIC QUESTIONS

Abstract

Material and Method The 15 questions commonly asked by patients undergoing orthodontic treatment were searched from the internet database and asked to ChatGPT3.5, ChatGPT4, BING, and ZekAI applications on May 15, 2023. The answers given by the applications were scored by two orthodontists, two orthodontic residents, and two dentists on a Likert Scale from 1 (most adequate) to 5 (most inadequate). Results In the study, two orthodontists (MGC, MU) and two orthodontic research assistants (AAA, ST) have assigned significantly lower scores to the ChatGPT4 application compared to the BING application (p<0.001). When the success scores assigned by the participants to the applications were evaluated, it was found that the BING application received significantly lower success scores from one of the dentists (ÇA) compared to the orthodontic research assistants (AAA, ST) (p<0.001). One of the dentists in the study (ÇA), on the other hand, evaluated the ZekAI application as significantly more successful compared to the orthodontist (MU), the other dentist (MMU), and the orthodontic research assistant (ST) (p<0.001). When all artificial intelligence chatbots were compared among themselves, it was found that ChatGPT4 provided the most desired responses (p<0.001). Conclusion This research has substantiated, based on the evaluation by participants, that the ChatGPT4 application represents the most reliable solution for assisting clinicians in addressing prevalent orthodontic queries, surpassing the ChatGPT3.5, BING, and ZekAI applications in this regard.

Keywords

• Artificial intelligence in orthodontics
• chatbots in orthodontics
• orthodontic patient education

YAPAY ZEKÂ DESTEKLİ SOHBET ROBOTLARININ YAYGIN ORTODONTİK SORULARI CEVAPLAMA BAŞARISININ DEĞERLENDİRİLMESİ

Abstract

Amaç Bu çalışmada ortodontik tedavi gören hastaların ortodontistlerine yaygın olarak sordukları soruların yapay zekâ destekli sohbet robotları tarafından cevaplanma başarısının değerlendirilmesi amaçlanmıştır. Gereç ve Yöntem Ortodontik tedavi gören hastaların yaygın olarak sordukları 15 soru internet veri tabanından araştırılarak ChatGPT3.5, ChatGPT4, BING ve ZekAI uygulamalarına 15 Mayıs 2023’te sorulmuştur. Uygulamaların verdikleri cevaplar iki ortodonti uzmanı, iki ortodonti araştırma görevlisi ve iki diş hekimi tarafından 1 (en yeterli)’den 5 (en yetersiz)’e kadar oluşturulan Likert Skalası ölçeğine göre puanlanmıştır. Bulgular İstatistiksel değerlendirmeler sonucunda çalışmada yer alan iki ortodonti uzmanı (MGC, MU) ve iki ortodonti asistanı (AAA, ST) ChatGPT4 uygulamasına BING uygulamasına göre istatistiksel olarak anlamlı derecede düşük puan vermiştir (p<0,001). Hekimlerin uygulamalara verdikleri başarı puanları değerlendirildiğinde, BING uygulamasının çalışmada yer alan diş hekimlerinden birinden (ÇA) aldığı başarı puanı ortodonti araştırma görevlilerine (AAA, ST) göre istatistiksel olarak anlamlı derecede düşük olmuştur (p<0,001). Çalışmada yer alan diş hekimlerinden biri (ÇA) ise ZekAI uygulamasını ortodonti uzmanı (MU), diğer diş hekimi (MMU) ve ortodonti araştırma görevlisine (ST) göre istatistiksel olarak anlamlı derecede başarılı değerlendirmiştir (p<0,001). Tüm yapay zekâ sohbet robotları kendi aralarında kıyaslandığında ChatGPT4’ün en istenilen cevapları verdiği bulunmuştur (p<0,001). Sonuç Bu çalışma, cevapları değerlendiren hekimlere göre ChatGPT4 uygulamasının ChatGPT3.5, BING ve ZekAI uygulamalarına göre yaygın ortodontik sorulara cevap vermede hekimlere yardımcı olabilecek en güvenilir uygulama olduğunu göstermiştir.

Keywords

• Ortodontide hasta bilgilendirme
• ortodontide yapay zekâ
• yapay zekâ sohbet robotu

References

1. 1. Russell S. Artificial intelligence a modern approach. Pearson Education; 2010.
2. 2. Wong S. Artificial intelligence in radiology: how will we be affected? European radiology. 2019;141-143. https://doi. org/10.1007/s00330-018-5644-3
3. 3. Andreu-Perez J, Deligianni F, Ravi D, Yang G. Artificial intelligence and robotics. arXiv preprint. 2018;1803.10813. https:// doi.org/10.48550/arXiv.1803.10813
4. 4. Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke and Vascular Neurology 2017;2:e000101. doi:10.1136/svn-2017- 000101
5. 5. Aggarwal A, Tam CC, Wu D, Li X, Qiao S. Artificial Intelligence– Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. Journal of Medical Internet Research. 2023;25:e40789. doi: 10.2196/40789.
6. 6. Lyons R, Arepallli S, Fromal O, Choi J, Jain N. Artificial intelligence chatbot performance in triage of ophthalmic conditions. Canadian Journal of Ophthalmology. 2023;S0008- 4182(23)00234-X. doi: 10.1016/j.jcjo.2023.07.016.
7. 7. Bhardwaz S, Kumar J. An Extensive Comparative Analysis of Chatbot Technologies-ChatGPT, Google BARD and Microsoft Bing. In 2nd International Conference on Applied Artificial Intelligence and Computing. India, 2023;673-679. doi: 10.1109/ ICAAIC56838.2023.10140214.
8. 8. ZekAI. Curiosity Technology [Internet]. Cited: 15 Mayıs 2023. Available from: https://zekai.co/tr/

9. 9. Skidmore K, Brook K, Thomson W, Harding W. Factors influencing treatment time in orthodontic patients. American Journal of Orthodontics and Dentofacial Orthopedics 2006;129(2):230-8. doi: 10.1016/j.ajodo.2005.10.003.
10. 10. Pratt M, Kluemper G, Lindstrom A. Patient compliance with orthodontic retainers in the postretention phase. American Journal of Orthodontics and Dentofacial Orthopedics. 2011;140(2):196- 201. doi: 10.1016/j.ajodo.2010.02.035.
11. 11. Gyawali R, Pokharel P, Giri J. Emergency appointments in orthodontics. APOS Trends Orthod. 2019;9(1):40-43. doi:10.25259/APOS-9-1-7
12. 12. Verma S, Sharma R, Deb S, Maitra D. Artificial intelligence in marketing: Systematic review and future research direction. International Journal of Information Management Data Insights. 2021;1(1):100002. https://doi.org/10.1016/j.jjimei.2020.100002
13. 13. Benichou L. Rôle de l'utilisation de l'intelligence artificielle ChatGPT dans la rédaction des articles scientifiques médicaux. The Role of Using ChatGPT AI in Writing Medical Scientific Articles. J Stomatol Oral Maxillofac Surg. 2023;124(5):101456. doi: 10.1016/j.jormas.2023.101456.
14. 14. de Cock C, Milne-Ives M, van Velthoven M, Alturkistani A, Lam C, Mein E. Effectiveness of conversational agents (virtual assistants) in health care: protocol for a systematic review. JMIR research protocols. 2020;9(3):e16934.
15. 15. Kessels R. Patients’ memory for medical information. Journal of the Royal Society of Medicine. 2003;96(5):219-222.
16. 16. Monill-Gonza´lez A, Rovira-Calatayud L, d'Oliveria N, Ustrell- Torrent J. Artificial intelligence in orthodontics: where are we now? A scoping review. Orthod Craniofac Res. 2021;24: 6-15.
17. 17. Ball M, Carla Smith N, Bakalar R. Personal health records: empowering consumers. J Healthc Inf Manag. 2007;21(1):77.
18. 18. Clarke M, Moore J, Steege L, Koopman R, Belden J, Canfield S. Health information needs, sources, and barriers of primary care patients to achieve patient-centered care: A literature review. Health inf. Health Informatics J. 2016;22(4):992-1016.
19. 19. Rinchuse D, Cozzani M. Effectiveness and efficiency in clinical orthodontic practice. International Orthodontics. 2015;13(4): 507–524.
20. 20. Perez-Pino A, Yadav S, Upadhyay M, Cardarelli L, Tadinada A. The accuracy of artificial intelligence–based virtual assistants in responding to routinely asked questions about orthodontics. The Angle Orthodontist. 2023;93(4):427-432.
21. 21. Balel Y. Can ChatGPT be used in oral and maxillofacial surgery? Journal of Stomatology, Oral and Maxillofacial Surgery. 2023;124(5):101471.
22. 22. Muttanahally K, Tadinada A, Mago J, Vyas R. Usefulness of Artificial Intelligence-based Virtual Assistants in Oral and Maxillofacial Radiology Report Writing. World Journal of Dentistry. 2021;12(2):97-102.
23. 23. Tustumi F, Andreollo N, Aguilar-Nascimento J. Future of the language models in healthcare: the role of chatGPT. Arquivos Brasileiros de Cirurgia Digestiva. 2023;356:e1721. doi: 10.1590/0102-672020230002e1727
24. 24. Nobles A, Leas E, Caputi T, Zhu S, Strathdee S, Ayers J. Responses to addiction help-seeking from Alexa, Siri, Google Assistant, Cortana, and Bixby intelligent virtual assistants. NPJ digital medicine. 2020; 3(1):11.