The role of Artificial Intelligence in Celiac disease support: analyzing ChatGPT's effectiveness for healthcare providers and patients

Abstract

Aims: Celiac disease is a significant autoimmune disorder that affects a substantial portion of the population, necessitating accurate and reliable information for effective management. Despite its importance, there are currently no studies assessing the performance of Artificial Intelligence (AI) language models, such as ChatGPT, in providing information on this condition. This study aims to evaluates the reliability and usefulness of ChatGPT's responses to frequently asked questions regarding celiac disease, thereby filling a critical gap in understanding the capabilities of AI in this important healthcare context. Methods: A total of 20 questions (10 for patients/caregivers and 10 for healthcare professionals) were prepared based on the most frequently searched queries about celiac disease using Google Trends. Responses generated by ChatGPT and scored by two independent Likert raters. Results: The analysis revealed strong inter-rater reliability, with Cronbach’s alpha values of α=0.839 for reliability and α=0.753 for usefulness, indicating robust agreement between raters. Notably, the highest reliability and usefulness scores for the patient and caregiver group were associated with questions on symptoms, the celiac disease diet, and gluten-free products. For the healthcare professionals group, key topics included diagnosis, pathological classification, and celiac disease comorbidities. Importantly, no significant differences were found between raters in the evaluation of reliability (p=0.939) and usefulness (p=0.102). Conclusion: This study demonstrates that AI-based language models like ChatGPT can serve as reliable and useful resources for both patients and healthcare professionals seeking information about celiac disease. While the model excelled in addressing commonly discussed topics, it revealed limitations in handling complex issues, emphasizing the need for ongoing refinement of AI tools. These findings support the integration of AI in healthcare communication, highlighting its potential to enhance access to crucial health information while underscoring the importance of continual improvement to meet diverse user needs.

Keywords

• Celiac disease
• chatGBT
• artificial intelligence

References

1. Al-Toma A, Volta U, Auricchio R, et al. European Society for the Study of Coeliac Disease (ESsCD) guideline for coeliac disease and other gluten-related disorders. United European Gastroenterol J. 2019;7(5):583-613. doi:10.1177/2050640619844125
2. Fasano A, Berti I, Gerarduzzi T, et al. Prevalence of celiac disease in at-risk and not-at-risk groups in the United States: a large multicenter study. Arch Intern Med. 2003;163(3):286-292. doi: 10.1001/archinte.163.3.286
3. Green PH, Cellier C. Celiac disease. N Engl J Med. 2007;357(17):1731-1743. doi:10.1056/NEJMra071600
4. Rubio-Tapia A, Hill ID, Kelly CP, Calderwood AH, Murray JA; American College of Gastroenterology. ACG clinical guidelines: diagnosis and management of celiac disease. Am J Gastroenterol. 2013;108(5):656-576; quiz 677. doi:10.1038/ajg.2013.79
5. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44-56. doi:10.1038/s41591-018-0300-7
6. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542(7639):115-118. doi:10.1038/nature21056
7. Palanica A, Flaschner P, Thommandram A, Li M, Fossat Y. Physicians' perceptions of chatbots in health care: cross-sectional web-based survey. J Med Internet Res. 2019;21(4):e12887. doi:10.2196/12887
8. Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020).

9. Jiang F, Jiang Y, Zhi H, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. 2017;2(4):230-243. doi:10.1136/svn-2017-000101
10. Kerbage A, Kassab J, El Dahdah J, et al. Accuracy of ChatGPT in common gastrointestinal diseases: impact for patients and providers. Clin Gastroenterol Hepatol. 2024;22(6):1323-1325.e3. doi:10.1016/j.cgh. 2023.11.008
11. Klang E, Sourosh A, Nadkarni GN, Sharif K, Lahat A. Evaluating the role of ChatGPT in gastroenterology: a comprehensive systematic review of applications, benefits, and limitations. Therap Adv Gastroenterol. 2023; 16:17562848231218618. doi:10.1177/17562848231218618
12. Rubio-Tapia A, Hill ID, Semrad C, et al. American College of gastroenterology guidelines update: diagnosis and management of celiac disease. Am J Gastroenterol. 2023;118(1):59-76. doi:10.14309/ajg. 0000000000002075
13. Uz C, Umay E. "Dr ChatGPT": is it a reliable and useful source for common rheumatic diseases? Int J Rheum Dis. 2023;26(7):1343-1349. doi:10.1111/1756-185X.14749
14. Green PH, Cellier C. Celiac disease. N Engl J Med. 2007;357(17):1731-1743. doi:10.1056/NEJMra071600
15. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44-56. doi:10.1038/s41591-018-0300-7
16. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542(7639):115-118. doi:10.1038/nature21056
17. Bot B, Suver C, Niederhoffer K. The use of artificial intelligence in medical literature review: a novel approach to evidence-based medicine. Med Hypotheses. 2017;105:1-3. doi:10.1016/j.mehy.2017.05.015
18. Jiang F, Jiang Y, Zhi H, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. 2017;2(4):230-243. doi:10.1136/svn-2017-000101
19. Nuti SV, Wayda B, Ranasinghe I, et al. The use of google trends in health care research: a systematic review. PLoS One. 2014;9(10):e109583. doi:10. 1371/journal.pone.0109583