DIGITAL CURIOSITY ABOUT FIBROMYALGIA: AN ANALYSIS OF THE QUALITY AND READABILITY OF ChatGPT’S RESPONSES BASED ON GOOGLE TRENDS

Abstract

Objective: The aim of this study is to evaluate the quality and readability of patient education materials about fibromyalgia generated by ChatGPT to assess whether this AI application can be reliably used as an online health information source for the general population.

Materials and Methods: In this study, Google Trends was used to identify the most frequently searched keywords related to fibromyalgia. The identified keywords were categorized into three groups: general information, diagnosis, and treatment, and were sequentially entered into ChatGPT. The quality of the generated responses was assessed using the modified DISCERN (mDISCERN) tool. The readability of the texts was evaluated using the Flesch-Kincaid Grade Level (FKGL) and Flesch-Kincaid Reading Ease (FRE) parameters.

Results: The mean mDISCERN score of the texts was 44.31±4.67, the mean FKGL score was 15.29±2.29, and mean FRE score was 24.23±10.27. The average mDISCERN score indicated that the texts were of moderate quality. According to the mean Flesch-Kincaid scores, the texts were found to be quite difficult to read and were mostly appropriate for readers at the undergraduate level or higher. Furthermore, the statistical analysis among the three categories revealed no significant differences in mDISCERN, FKGL, and FKRE scores.

Conclusion: AI language models like ChatGPT hold innovative potential in healthcare communication by providing accessible and unbiased information that can support health literacy and patient education. However, the high level of reading comprehension required currently limits their effectiveness and reliability for direct patient use. Therefore, the content generated by ChatGPT should be considered a complementary resource and should be critically examined by users.

Keywords

• Fibromyalgia
• ChatGPT
• Readability and Quality Assessment

FİBROMİYALJİYE DAİR DİJİTAL MERAK: ChatGPT’NİN GOOGLE TRENDLER TABANLI YANITLARININ KALİTE VE OKUNABİLİRLİK ANALİZİ

Abstract

Amaç: Bu çalışmanın amacı, ChatGPT tarafından fibromiyalji hakkında oluşturulan hasta eğitim materyallerinin kalitesini ve okunabilirliğini değerlendirmek ve bu yapay zeka uygulamasının genel popülasyon için güvenilir bir çevrimiçi sağlık bilgi kaynağı olarak kullanılıp kullanılamayacağını değerlendirmektir.

Gereç ve Yöntemler: Çalışmada, fibromiyalji ile ilgili en sık aranan anahtar kelimeler Google Trendler kullanılarak belirlendi. Belirlenen anahtar kelimeler genel bilgi, tanı ve tedavi olmak üzere üç gruba ayrıldı ve sırayla ChatGPT'ye girildi. Oluşturulan yanıtların kalitesi, modifiye edilmiş DISCERN (mDISCERN) aracı kullanılarak değerlendirildi. Metinlerin okunabilirliği, Flesch-Kincaid Sınıf Düzeyi (FKGL) ve Flesch-Kincaid Okuma Kolaylığı (FRE) parametreleri kullanılarak değerlendirildi.

Bulgular: Metinlerin ortalama mDISCERN puanı 44.31±4.67, ortalama FKGL puanı 15.29±2.29 ve ortalama FRE puanı 24.23±10.27 olarak bulundu. Ortalama mDISCERN puanı, metinlerin orta kalitede olduğunu gösterdi. Ortalama Flesch-Kincaid skorlarına göre, metinlerin oldukça zor okunabilir olduğu ve çoğunlukla lisans düzeyi veya üzeri okuma becerisi gerektirdiği belirlendi. Ayrıca, üç kategori arasındaki istatistiksel analiz, mDISCERN, FKGL ve FKRE puanlarında anlamlı bir fark ortaya koymadı.

Sonuç: Özetle, ChatGPT gibi yapay zekâ dil modelleri, sağlık iletişiminde erişilebilir ve tarafsız bilgi sunarak sağlık okuryazarlığını ve hasta eğitimini destekleme açısından yenilikçi bir potansiyele sahiptir. Ancak, yüksek düzeyde okuma-anlama becerisi gerektirmesi, doğrudan hasta kullanımı açısından etkinliğini ve güvenilirliğini sınırlamaktadır. Bu nedenle, ChatGPT tarafından üretilen içerikler tamamlayıcı bir kaynak olarak değerlendirilmelidir ve kullanıcılar tarafından eleştirel biçimde incelenmelidir.

Keywords

• Fibromiyalji
• ChatGPT
• Okunabilirlik ve Kalite Değerlendirmesi

References

1. Filipovic T, Filipović A, Nikolic, Gimigliano F, Stevanov J, Bosanac I. Fibromyalgia: Understanding, diagnosis and modern approaches to treatment. J Clin Med 2025;14(3):955. google scholar
2. Goldenberg DL, Kaplan M. Fibromyalgia: Treatment in adults. In: Connor RF, editor. UpToDate. Wolters Kluwer; 2025. google scholar
3. Golan R, Reddy R, Ramasamy R. The rise of artificial intelligence-driven health communication. Transl Androl Urol 2024;13(2):356-8. google scholar
4. Behers BJ, Stephenson-Moe CA, Gibons RM, Vargas IA, Wojtas CN, Rosario MA, et al. Assessing the quality of patient education materials on cardiac catheterization from artificial intelligence chatbots: An observational cross-sectional study. Cureus 2024;16(9):e69996. google scholar
5. Günay AE, Özer A, Yazıcı A, Sayer G. Comparison of ChatGPT versions in informing patients with rotator cuff injuries. JSES Int 2024;8(5):1016-8. google scholar
6. Ozduran E, Hancı V, Erkin Y, Özbek İC, Abdulkerimov V. Assessing the readability, quality and reliability of responses produced by ChatGPT, Gemini, and Perplexity regarding most frequently asked keywords about low back pain. PeerJ 2025;13:e18847. google scholar
7. Gibson D, Jackson S, Shanmugasundaram R, Seth I, Siu A, Ahmadi N, et al. Evaluating the efficacy of ChatGPT as a patient education tool in prostate cancer: Multimetric assessment. J Med Internet Res 2024;26:e55939. google scholar
8. Tian Tran J, Burghall A, Blydt-Hansen T, Cammer A, Goldberg A, Hamiwka L, et al. Exploring the ability of ChatGPT to create quality patient education resources about kidney transplant. Patient Educ Couns 2024;129:108400. google scholar

9. Vallurupalli M, Shah ND, Vyas RM. Optimizing readability of patient-facing hand surgery education materials using Chat Generative Pretrained Transformer (ChatGPT) 3.5. J Hand Surg Am 2024;49(10):986-91. google scholar
10. Peng S, Wang D, Liang Y, Xiao W, Zhang Y, Liu L. AI-ChatGPT/GPT-4: An booster for the development of physical medicine and rehabilitation in the new era. Ann Biomed Eng 2024;52(3):462–6. google scholar
11. Ghanem YK, Rouhi AD, Al-Houssan A, Saleh Z, Moccia MC, Joshi H, et al. Dr. Google to Dr. ChatGPT: Assessing the content and quality of artificial intelligence-generated medical information on appendicitis. Surg Endosc 2024;38(5):2887-93. google scholar
12. D'Ambrosi R, Annibaldi A, Carrozzo A, Vieira TD, An JS, Freychet B, et al. Evaluating the reliability of YouTube as a source of information for meniscal ramp lesions. Orthop J Sports Med 2024;12(1):23259671231219815. google scholar
13. Flesch R. A new readability yardstick. J Appl Psychol 1948;32(3):221-33. google scholar
14. Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel. Res Branch Rep 1975;8-75. google scholar
15. Guven Y, Ozdemir OT, Kavan MY. Performance of artificial intelligence chatbots in responding to patient queries related to traumatic dental injuries: A comparative study. Dent Traumatol 2025;41(3):338-47. google scholar
16. Huang J, Galal G, Etemadi M, Vaidyanathan M. Evaluation and mitigation of racial bias in clinical machine learning models: Scoping review. JMIR Med Inform 2022;10(5):e36388. google scholar
17. Yang Y, Liu X, Jin Q, Huang F, Lu Z. Unmasking and quantifying racial bias of large language models in medical report generation. Commun Med 2024;4(1):176. google scholar
18. Rooney MK, Santiago G, Perni S, Horowitz DP, McCall AR, Einstein AJ, et al. Readability of patient education materials from high-impact medical journals: A 20-year analysis. J Patient Exp 2021;8:2374373521998847. google scholar
19. Badarudeen S, Sabharwal S. Assessing readability of patient education materials: Current role in orthopaedics. Clin Orthop Relat Res 2010;468(10):2572–80. google scholar
20. Özbek İC, Hancı V, Özduran E. Digital Guidance: Quality and readability analysis of artificial intelligence-generated spondyloarthropathy texts. Turk J Osteoporos 2025;31(1):12-18. google scholar
21. Rouhi AD, Ghanem YK, Yolchieva L, et al. Can Artificial intelligence improve the readability of patient education materials on aortic stenosis? A Pilot Study. Cardiol Ther 2024;13(1):137-147. google scholar
22. Ayre J, Mac O, McCaffery K, McKay BR, Liu M, Shi Y, et al. New frontiers in health literacy: Using ChatGPT to simplify health information for people in the community. J Gen Intern Med 2024;39(4):573-7. google scholar