Readability and Appropriateness of Responses Generated by ChatGPT 3.5, Gemini 2.5, and Microsoft Copilot to Frequently Asked Questions About Central Serous Chorioretinopathy
Abstract
Background: This study aims to compare the readability levels and informational quality of responses generated by three different artificial intelligence (AI)-based text generation models in relation to Central Serous Chorioretinopathy (CSCR).
Materials and Methods: A total of 40 questions pertaining to CSCR were formulated based on articles indexed in PubMed over the past ten years. These questions were submitted to three AI-based text generation tools ChatGPT 3.5, Gemini 2.5, and Microsoft Copilot. The resulting responses were analyzed for readability using five standard indices: Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), Gunning Fog Index (GFOG), and Automated Readability Index (ARI). Sentence lengths and structural complexity were also assessed. Content
quality was independently evaluated and scored by two researchers using a standardized rubric.
Results: Nor adequate readability. A statistically significant difference in FRES scores was observed among the models (p = 0.01). Similarly, none of the models met the acceptable readability standards across the other four indices, with all scores exceeding recommended limits indicating generally poor readability. Among the tools, Gemini yielded significantly higher quality scores compared to the others (p < .001), suggesting superior informational content. Conversely, Microsoft Copilot produced more concise outputs characterized by shorter and fewer sentencesone of the models achieved the
optimal FRES threshold (≥ 60) required f.
Conclusions: The findings suggest that AI-generated responses regarding CSCR are often overly technical and may not be easily comprehensible to individuals without a medical background. Moreover, the study highlights variability among different AI models in terms of both readability and content quality. These results underscore the importance of critically evaluating AI-generated medical content prior to its dissemination for public or clinical use.
Keywords
Central Serous Chorioretinopathy, Readability, Content Quality, Artificial Intelligence
Supporting Institution
The authors received no financial support for the research, authorship, and/or publication of this article.
Ethical Statement
This study did not involve human participants, patient data, or animal subjects. The study was based on the analysis of responses generated by artificial intelligence chatbots to predefined questions about Central Serous Chorioretinopathy. Therefore, ethics committee approval and informed consent to participate were not required.
Thanks
Not applicable.
References
- Olszewski R, Watros K, Mańczak M, Owoc J, Jeziorski K, Brzeziński J. Assessing the response quality and readability of chatbots in cardiovascular health, oncology, and psoriasis: A comparative study. Int J Med Inform. 2024;190:105562.
- Bohr A, Memarzadeh K. The rise of artificial intelligence in healthcare applications. In: Artificial Intelligence in Healthcare. 2020. p. 25–60.
- Nirala KK, Singh NK, Purani VS. A survey on providing customer and public administration based services using AI: chatbot. Multimed Tools Appl. 2022;81:22215–22246.
- Semeraro F, Gamberini L, Carmona F, Monsieurs KG. Clinical questions on advanced life support answered by artificial intelligence: A comparison between ChatGPT, Google Bard and Microsoft Copilot. Resuscitation. 2024;195:110114.
- Diamond C, Rundle CW, Albrecht JM, Nicholas MW. Chatbot utilization in dermatology: A potential amelioration to burnout in dermatology. Dermatol Online J. 2022;28(5):13030.
- Yan S, Du D, Liu X, Dai Y, Kim MK, Zhou X, et al. Assessment of the reliability and clinical applicability of ChatGPT’s responses to patients’ common queries about rosacea. Patient Prefer Adherence. 2024;18:249–253.
- Young JN, Ross O, Poplausky D, et al. The utility of ChatGPT in generating patient-facing and clinical responses for melanoma. J Am Acad Dermatol. 2023;89(3):602–604.
- Musheyev D, Pan A, Loeb S, Kabarriti AE. How well do artificial intelligence chatbots respond to the top search queries about urological malignancies? Eur Urol. 2024;85(1):13–16.
- Pan A, Musheyev D, Bockelman D, Loeb S, Kabarriti AE. Assessment of artificial intelligence chatbot responses to top searched queries about cancer. JAMA Oncol. 2023;9(10):1437–1440.
- Fung AT, Yang Y, Kam AW. Central serous chorioretinopathy: A review. Clin Exp Ophthalmol. 2023;51(3):243–270.