The Promise and the Challenge: Large Language Models for Patient Education - Are We There Yet?

Abstract

This study aimed to evaluate the potential of large language models (LLMs) for delivering patient education materials. Two LLMs, Gemini and ChatGPT 3.5, were analysed for their ability to provide clear and understandable information on the topic of blepharitis, a common eye condition. The understandability and actionability of the information provided by the LLMs in response to a set of questions were evaluated using PEMAT, a standardised tool for assessing educational materials. The responses included the important aspects of blepharitis, yet the Flesch-Kincaid readability scores were below the suggested range of 60-70 for patient education materials. Gemini received a score of 38.75, whereas ChatGPT 3.5 earned 26.35, suggesting that the content might be too intricate for the target audience.. These findings suggest that while LLMs have the potential to be informative resources, their current readability levels may limit their effectiveness in providing accessible health information to patients. Further research is needed to explore methods for adapting LLM outputs to ensure clear and concise communication suitable for patient education.

Keywords

• Large Language Models (LLMs)
• Patient Education
• Readability
• Blepharitis
• Natural Language Processing (NLP)
• Health Communication

Hasta Bilgilendirmesinde Yapay Zeka Robotları: Hazır mı?

Abstract

Bu çalışma, hasta bilgilendirilmesinde büyük dil modellerinin (LLM'ler) potansiyelini değerlendirmeyi amaçladı. Gemini ve ChatGPT 3.5 adlı iki LLM, yaygın ve kronik bir göz rahatsızlığı olan blefarit konusunda net ve anlaşılır bilgi sağlama yetenekleri açısından analiz edildi. LLM'ler tarafından bir dizi soruya yanıt olarak sağlanan bilgilerin anlaşılabilirliği ve eyleme geçirilebilirliği, eğitim materyallerini değerlendirmek için standartlaştırılmış bir araç olan PEMAT kullanılarak değerlendirildi. Yanıtlar blefaritin önemli yönlerini içeriyordu, ancak Flesch-Kincaid okunabilirlik skorları hasta eğitim materyalleri için önerilen 60-70 aralığının altındaydı. Gemini 38,75 puan alırken, ChatGPT 3.5 26,35 puan aldı, bu da içeriğin hedef kitle için çok karmaşık olabileceğini düşündürdü. Bu bulgular, LLM'lerin bilgilendirici kaynaklar olma potansiyeline sahip olmasına rağmen, mevcut okunabilirlik seviyelerinin hastalara erişilebilir sağlık bilgisi sağlamadaki etkinliklerini sınırlayabileceğini göstermektedir. Hasta eğitimine uygun açık ve özlü iletişimi sağlamak için LLM çıktılarını uyarlama yöntemlerini araştırmak için daha fazla araştırmaya ihtiyaç vardır.

Keywords

• Büyük Dil Modelleri (LLM)
• Hasta Eğitimi
• Okunabilirlik
• Blefarit
• Doğal Dil İşleme (NLP)
• Sağlık İletişimi

References

1. 1. Eberhardt M, Rammohan G. Blepharitis. In: StatPearls. Treasure Island (FL): StatPearls Publishing; January 23, 2023.
2. 2. Trattler W, Karpecki P, Rapoport Y, et al. The Prevalence of Demodex Blepharitis in US Eye Care Clinic Patients as Determined by Collarettes: A Pathognomonic Sign. Clinical Ophthalmology 2022;16; doi: 10.2147/OPTH.S354692.
3. 3. Pflugfelder SC, Karpecki PM, Perez VL. Treatment of blepharitis: Recent clinical trials. Ocular Surface 2014;12(4):273–284; doi: 10.1016/j.jtos.2014.05.005.
4. 4. Duncan K, Jeng BH. Medical Management of Blepharitis. Curr Opin Ophthalmol 2015;26(4); doi: 10.1097/ICU.0000000000000164.
5. 5. Shah PP, Stein RL, Perry HD. Update on the Management of Demodex Blepharitis. Cornea 2022;41(8); doi: 10.1097/ICO.0000000000002911.
6. 6. Nichols KK, Foulks GN, Bron AJ, et al. The International Workshop on Meibomian Gland Dysfunction: Executive Summary. Investigative Opthalmology & Visual Science 2011;52(4):1922; doi: 10.1167/iovs.10-6997a.
7. 7. Rhodes L, Huisingh C, McGwin G, et al. Eye Care Quality and Accessibility Improvement in the Community (EQUALITY): impact of an eye health education program on patient knowledge about glaucoma and attitudes about eye care. Patient Relat Outcome Meas 2016; doi: 10.2147/prom.s98686.
8. 8. Görtz M, Baumgärtner K, Schmid T, et al. An artificial intelligence-based chatbot for prostate cancer education: Design and patient evaluation study. Digit Health 2023;9. doi: 10.1177/20552076231173304.

9. 9. Frangoudes F, Hadjiaros M, Schiza EC, et al. An Overview of the Use of Chatbots in Medical and Healthcare Education. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 2021;12785 LNCS:170–184; doi: 10.1007/978-3-030-77943-6_11/cover.
10. 10. Chaix B, Bibault JE, Pienkowski A, et al. When chatbots meet patients: One-year prospective study of conversations between patients with breast cancer and a chatbot. JMIR Cancer 2019;5(1); doi: 10.2196/12856.
11. 11. PEMAT for Printable Materials (PEMAT-P) | Agency for Healthcare Research and Quality. Available from: https://www.ahrq.gov/health-literacy/patient-education/pemat-p.html.
12. 12. Biggs J, Collis K. Evaluating the Quality of Learning: The SOLO Taxonomy (Structure of the Observed Learning Outcome). 2014.
13. 13. Kincaid PJ. Derivation of New Readability Formulas. Naval Technical Training Command Millington TN Research Branch 1975;(February).
14. 14. Cohen J. A Coefficient of Agreement for Nominal Scales. Educ Psychol Meas 1960;20(1); doi: 10.1177/001316446002000104.
15. 15. Sallam M. ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare (Switzerland) 2023;11(6); doi: 10.3390/healthcare11060887.
16. 16. Zheng Y, Wang L, Feng B, et al. Innovating Healthcare: The Role of ChatGPT in Streamlining Hospital Workflow in the Future. Ann Biomed Eng 2023; doi: 10.1007/S10439-023-03323-W.
17. 17. Cascella M, Montomoli J, Bellini V, et al. Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios. J Med Syst 2023;47(1); doi: 10.1007/S10916-023-01925-4.
18. 18. Laranjo L, Dunn AG, Tong HL, et al. Conversational agents in healthcare: A systematic review. Journal of the American Medical Informatics Association 2018;25(9):1248–1258; doi: 10.1093/jamia/ocy072.
19. 19. Car LT, Dhinagaran DA, Kyaw BM, et al. Conversational agents in health care: Scoping review and conceptual analysis. J Med Internet Res 2020;22(8); doi: 10.2196/17158.
20. 20. Ayers JW, Zhu Z, Poliak A, et al. Evaluating Artificial Intelligence Responses to Public Health Questions. JAMA Netw Open 2023;6(6); doi: 10.1001/jamanetworkopen.2023.17517.
21. 21. Wu Q, Jiang S. The Effects of Patient-Centered Communication on Emotional Health: Examining the Roles of Self-Efficacy, Information Seeking Frustration, and Social Media Use. J Health Commun 2023;28(6):349–359; doi: 10.1080/10810730.2023.2208537.
22. 22. Bhirud N, Tatale S, Randive S, et al. A Literature Review On Chatbots In Healthcare Domain. 2019.
23. 23. Lin J, Joseph T, Parga-Belinkie JJ, et al. Development of a practical training method for a healthcare artificial intelligence (AI) chatbot. BMJ Innov 2021;7(2):441–444; doi: 10.1136/bmjinnov-2020-000530.
24. 24. Multilingualism and WHO. n.d. Available from: https://www.who.int/about/policies/multilingualism.
25. 25. Crossing the Quality Chasm: A New Health System for the 21st Century. 2001; doi: 10.17226/10027.
26. 26. Li H, Moon JT, Purkayastha S, et al. Ethics of large language models in medicine and medical research. Lancet Digit Health 2023;5(6):e333–e335; doi: 10.1016/S2589-7500(23)00083-3.