Evaluating ChatGPT as a Patient Education Tool: Insights on Quality, Readability, and Reliability for Trigger Finger FAQs

Öz

Aim: Trigger finger (TF), or stenosing tenosynovitis, causes pain, snapping, and finger locking. It greatly affects patients' quality of life, prompting frequent inquiries to healthcare providers. ChatGPT, an AI language model, has gained popularity as a tool for patient education. This study evaluated the quality, readability, and usability of ChatGPT’s responses to common TF FAQs. Methods A set of FAQs regarding TF was developed based on reputable sources such as WebMD, Mayo Clinic, and NHS Trusts. Two experienced surgeons reviewed and refined the questions before submitting them to ChatGPT-4 for response generation. The quality of the responses was evaluated using the Global Quality Score (GQS) and DISCERN scale, while readability was assessed using the Flesch Reading Ease Score (FRES) and Flesch–Kincaid Grade Level (FKGL). Inter-rater reliability was determined using Cohen’s Kappa. Results: Twenty responses were evaluated, yielding a mean GQS of 3.8, indicating moderate to high quality (SD = 0.71). DISCERN scores averaged 37.08 ± 7.64, reflecting fair to good quality. Readability analysis showed a FRES score of 43.40, suggesting the content is challenging for those without a college education. The mean FKGL was 12.17, indicating advanced reading requirements. Prognosis-related questions had better readability scores than treatment-related responses, which were more complex. Conclusion: ChatGPT shows promise for patient education with moderate to high-quality responses about TF. However, advanced reading levels may limit wider accessibility. Improving readability and tailoring responses to diverse needs are vital for effectiveness. Human oversight is essential to ensure accuracy and usability of AI-generated content.

Anahtar Kelimeler

• Trigger finger
• ChatGPT
• Patient education
• Readability

Kaynakça

1. Makkouk AH, Oetgen ME, Swigart CR, Dodds SD. Trigger finger: etiology, evaluation, and treatment. Curr Rev Musculoskelet Med. 2008;1:92–6.
2. Langer D, Maeir A, Michailevich M, Applebaum Y, Luria S. Using the international classification of functioning to examine the impact of trigger finger. Disabil Rehabil. 2016;38(26):2530–7.
3. El-halawany EF, Yamany AAER, Gaballah SAA, Atta HK. Effect of dry needling on quality of life in patients with trigger finger: a randomized controlled trial. Bull Fac Phys Ther. 2024;29(1):11.
4. Dala-Ali BM, Nakhdjevani A, Lloyd MA, Schreuder FB. The efficacy of steroid injection in the treatment of trigger finger. Clin Orthop Surg. 2012;4(4):263–8.
5. Shakeel H, Ahmad TS. Steroid injection versus NSAID injection for trigger finger: a comparative study of early outcomes. J Hand Surg Am. 2012;37(7):1319–23.
6. Rao AJ, Dy CJ, Goldfarb CA, Cohen MS, Wysocki RW. Patient preferences and utilization of online resources for patients treated in hand surgery practices. Hand. 2019;14(2):277–83.
7. AlShenaiber A, Datta S, Mosa AJ, Binhammer PA, Ing EB. Large Language Models in the Diagnosis of Hand and Peripheral Nerve Injuries: An Evaluation of ChatGPT and the Isabel Differential Diagnosis Generator. J Hand Surg Glob Online. 2024;6(6):847–54.
8. Leypold T, Schäfer B, Boos AM, Beier JP. Artificial Intelligence-Powered Hand Surgery Consultation: GPT-4 as an Assistant in a Hand Surgery Outpatient Clinic. J Hand Surg Am. 2024;49(11):1078–88.

9. Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. J Med Syst. 2023;47(1):33.
10. OpenAI TB. Chatgpt: Optimizing language models for dialogue. OpenAI. 2022.
11. Gunduz ME, Matis GK, Ozduran E, Hanci V. Evaluating the Readability, Quality, and Reliability of Online Patient Education Materials on Spinal Cord Stimulation. Turk Neurosurg. 2024;34(4).
12. Flesch R. A new readability yardstick. J Appl Psychol. 1948;32(3):221.
13. Simpson D. The readability test tool. Access 21/10/2011. http://www.read-able.com ; 2013.
14. Bw H. Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey. Arch Intern Med. 2005;165:2618–24.
15. Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C, Mishkin P, et al. Training language models to follow instructions with human feedback. Adv Neural Inf Process Syst. 2022;35:27730–44.
16. Haidar O, Jaques A, McCaughran PW, Metcalfe MJ. AI-Generated Information for Vascular Patients: Assessing the Standard of Procedure-Specific Information Provided by the ChatGPT AI-Language Model. Cureus. 2023;15(11).
17. Onder CE, Koc G, Gokbulut P, Taskaldiran I, Kuskonmaz SM. Evaluation of the reliability and readability of ChatGPT-4 responses regarding hypothyroidism during pregnancy. Sci Rep. 2024;14(1):243.
18. Li LT, Sinkler MA, Adelstein JM, Voos JE, Calcei JG. ChatGPT responses to common questions about anterior cruciate ligament reconstruction are frequently satisfactory. Arthrosc J Arthrosc Relat Surg. 2024.
19. Tan RES, Teo WZW, Puhaindran ME. Artificial Intelligence in Hand Surgery–How Generative AI is Transforming the Hand Surgery Landscape. J Hand Surg (Asian-Pacific Vol). 2024;29(02):81–7.
20. Leypold T, Schäfer B, Boos A, Beier JP. Can AI think like a plastic surgeon? Evaluating GPT-4’s clinical judgment in reconstructive procedures of the upper extremity. Plast Reconstr Surgery–Global Open. 2023;11(12):e5471.