Readability, reliability, and quality of AI chatbot responses on home mechanical ventilation

Ferhan Demirer Aydemir; Volkan Hancı

Readability, reliability, and quality of AI chatbot responses on home mechanical ventilation

Abstract

Aims: Home mechanical ventilation (HMV) is an essential component of long-term respiratory care. Patients and caregivers increasingly use online resources, including Artificial Intelligence (AI)-based chatbots, to obtain information regarding HMV. However, the readability, reliability, and quality of AI-generated information on HMV remain unclear. This study aimed to evaluate the readability, reliability, and quality of responses generated by ChatGPT, Gemini, and Perplexity for frequently asked questions about HMV. Methods: A cross-sectional content analysis was conducted in December 2025 using 47 high-interest HMV-related questions identified via Google Trends. Each question was entered verbatim into ChatGPT, Gemini, and Perplexity under standardized conditions. Readability was assessed using seven validated indices. Reliability was evaluated using the JAMA benchmark criteria and the modified DISCERN instrument, while overall educational quality was assessed using the Global Quality Score (GQS) and the Ensuring Quality Information for Patients (EQIP) tool. Non-parametric statistical tests were applied for inter-model comparisons. Results: All AI-generated responses significantly exceeded the recommended sixth-grade readability threshold across all indices (p<0.001). ChatGPT produced comparatively lower grade-level readability scores, whereas Perplexity demonstrated significantly higher reliability and quality scores. Gemini showed intermediate performance across most metrics. Significant inter-model differences were observed for multiple readability, reliability, and quality measures. A consistent association was identified between higher informational quality scores and increased textual complexity. Conclusion: AI-generated information on HMV is generally more complex than recommended for patients. Differences between platforms suggest that AI-generated medical content is not interchangeable and should be used only as a supplement to professional medical guidance.

Keywords

Supporting Institution

This study did not receive any financial support.

Ethical Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by the Non-Interventional Clinical Research Ethics Committee of Çanakkale Onsekiz Mart University (Approval No: 2025-402; 07 January 2026).

Thanks

None.

References

Simonds AK. Home mechanical ventilation: an overview. Ann Am Thorac Soc. 2016;13(11):2035-2044. doi:10.1513/AnnalsATS.201606-454FR
MacIntyre EJ, Asadi L, McKim DA, Bagshaw SM. Clinical outcomes associated with home mechanical ventilation: a systematic review. Can Respir J. 2016;2016:6547180. doi:10.1155/2016/6547180
Płaszewska-Żywko L, Fajfer-Gryz I, Cichoń J, Kózka M. Burden, social support, and coping strategies in family caregivers of individuals receiving home mechanical ventilation: a cross-sectional study. BMC Nurs. 2024;23(1):346. doi:10.1186/s12912-024-02024-6
McKim DA, Road J, Avendano M, et al. Home mechanical ventilation: a Canadian Thoracic Society clinical practice guideline. Can Respir J. 2011;18(4):197-215. doi:10.1155/2011/139769
Lipprandt M, Liedtke W, Langanke M, Klausen A, Baumgarten N, Röhrig R. Causes of adverse events in home mechanical ventilation: a nursing perspective. BMC Nurs. 2022;21(1):264. doi:10.1186/s12912-022-01038-2
Clark M, Bailey S. Chatbots in health care: connecting patients to information: emerging health technologies. Ottawa (ON): Canadian Agency for Drugs and Technologies in Health; January 2024.
Iqbal U, Tanweer A, Rahmanti AR, Greenfield D, Lee LTJ, Li YCJ. Impact of large language model (ChatGPT) in healthcare: an umbrella review and evidence synthesis. J Biomed Sci. 2025;32(1):45. doi:10.1186/s12929-025-01131-z
Huo B, Boyle A, Marfo N, et al. Large language models for chatbot health advice studies: a systematic review. JAMA Netw Open. 2025;8(2): e2457879. doi:10.1001/jamanetworkopen.2024.57879

Fahy S, Oehme S, Milinkovic D, Jung T, Bartek B. Assessment of quality and readability of information provided by ChatGPT in relation to anterior cruciate ligament injury. J Pers Med. 2024;14(1):104. doi:10.3390/jpm14010104
Hanci V, Ergün B, Gül Ş, et al. Assessment of readability, reliability, and quality of ChatGPT®, Bard®, Gemini®, Copilot®, and Perplexity® responses on palliative care. Medicine (Baltimore). 2024;103(33):e39305. doi:10.1097/MD.0000000000039305
Ozduran E, Hanci V. Evaluation of the 100 most cited articles on Sjogren's syndrome: a bibliometric analysis. J Basic Clin Health Sci. 2023;7:11-20. doi:10.30621/jbachs.1096610
Ozduran E. YouTube as an information source for myofascial pain syndrome. In: Unlu Z, editor. Current approaches to myofascial pain syndrome. 1st ed. Ankara: Turkiye Klinikleri; 2026:p.377-382. doi:10.5336/978-625-395-854-1_p377
Ozduran E, Hanci V. Exploring the reliability and accessibility of YouTube for cerebral palsy rehabilitation information. J Child. 2024;24: 15-24. doi:10.26650/jchild.2024.1301406
Ozduran E, Hanci V. YouTube as a source of information about pulmonary rehabilitation during the COVID-19 pandemic. Eurasian J Pulmonol. 2022;24:107-114. doi:10.14744/ejp.2022.9221
Ozbek IC, Ozduran E, Hanci V. Digital guidance: quality and readability analysis of artificial intelligence-generated spondyloarthropathy texts. Turk J Osteoporos. 2025;31:12-18. doi:10.4274/tod.galenos.2024.76743
Mengi A, Ozduran E. Evaluating the readability, quality, and reliability of responses generated by artificial intelligence programs about acupuncture for pain. J Physical Med Rehab Sci. 2026;29:245-253. doi:10.31609/jpmrs.2025-115285
Ozduran E, Hanci V. Digital rehabilitation in Parkinson's disease: the role of Artificial Intelligence-assisted exercise training. Turk J Osteoporos. 2025. doi:10.4274/tod.galenos.2025.66664
Ozduran E, Hanci V. Evaluating the readability, quality, and reliability of online information on Sjogren's syndrome. Indian J Rheumatol. 2023; 18:16-25. doi:10.4103/injr.injr_56_22
Li J, Dada A, Puladi B, Kleesiek J, Egger J. ChatGPT in healthcare: a taxonomy and systematic review. Comput Methods Programs Biomed. 2024;245:108013. doi:10.1016/j.cmpb.2024.108013
Wei Q, Yao Z, Cui Y, Wei B, Jin Z, Xu X. Evaluation of ChatGPT-generated medical responses: a systematic review and meta-analysis. J Biomed Inform. 2024;151:104620. doi:10.1016/j.jbi.2024.104620
Daram NR, Maxwell RA, D’Amato J, Massengill JC. Can Artificial Intelligence improve the readability of patient education information in gynecology? Am J Obstet Gynecol. 2025;233(6):640.e1-640.e9. doi:10.1016/j.ajog.2025.06.047
Shukla IY, Sun MZ. Online and ChatGPT-generated patient education materials regarding brain tumor prognosis fail to meet readability standards. J Clin Neurosci. 2025;138:111410. doi:10.1016/j.jocn.2025.111410
Stephenson-Moe CA, Behers BJ, Gibons RM, et al. Assessing the quality and readability of patient education materials on chemotherapy cardiotoxicity from Artificial Intelligence chatbots: an observational cross-sectional study. Medicine (Baltimore). 2025;104(15):e42135. doi:10.1097/MD.0000000000042135
Nelson HC, Beauchamp MT, Pace AA. The reliability gap: how traditional search engines outperform Artificial Intelligence chatbots in rosacea public health information quality. Cureus. 2025;17(6):e86543. doi:10.7759/cureus.86543
Altom DS, Taha AIA, Hussein AAAM, et al. Artificial Intelligence-based chatbots in chronic disease management: a systematic review of applications and challenges. Cureus. 2025;17(3):e81001. doi:10.7759/cureus.81001
Khankeh HR, Ebadi A, Norouzi Tabrizi K, Moradian ST. Home health care for mechanical ventilation-dependent patients: a grounded theory study. Health Soc Care Community. 2022;30(5):e2157-e2168. doi:10.1111/hsc.13654
Wasilewski MB, Kokorelias KM, Nonoyama M, et al. The experience of family caregivers of ventilator-assisted individuals who participated in a pilot web-based peer support program: a qualitative study. Digit Health. 2022;8:20552076221134964. doi:10.1177/20552076221134964
Kara M, Ozduran E, Kara MM, Ozbek IC, Hanci V. Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about ankylosing spondylitis. PLoS One. 2025;20(6):e0326351. doi:10.1371/journal.pone.0326351
Ozduran E. Evaluation of the readability of Turkish internet-based patient education materials related to low back pain. J Dokuz Eylul Univ Med Fac. 2022;36:135-150. doi:10.18614/deutip.1174522
Ayers JW, Poliak A, Dredze M, et al. Comparing physician and Artificial Intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med. 2023;183(6):589-596. doi:10.1001/jamainternmed.2023.1838
Traylor DO, Kern KV, Anderson EE, Henderson R. Beyond the screen: the impact of generative Artificial Intelligence on patient learning and the patient-physician relationship. Cureus. 2025;17(1):e76825. doi:10.7759/cureus.76825
Daraz L, Morrow AS, Ponce OJ, et al. Can patients trust online health information? A meta-narrative systematic review addressing the quality of health information on the internet. J Gen Intern Med. 2019;34(9):1884-1891. doi:10.1007/s11606-019-05109-0
Ozduran E, Akkoc I, Büyükçoban S, Erkin Y, Hanci V. Readability, reliability and quality of responses generated by ChatGPT, Gemini, and Perplexity for the most frequently asked questions about pain. Medicine (Baltimore). 2025;104(11):e41780. doi:10.1097/MD.0000000000041780
Ömür Arça D, Erdemir İ, Kara F, et al. Assessing the readability, reliability, and quality of Artificial Intelligence chatbot responses to the 100 most searched queries about cardiopulmonary resuscitation: an observational study. Medicine (Baltimore). 2024;103(22):e38352. doi:10.1097/MD.0000000000038352
Aggarwal S, Sridhar J, John VP, et al. An observational study to evaluate readability and reliability of AI-generated brochures for emergency medical conditions. Cureus. 2024;16(8):e68307. doi:10.7759/cureus.68307
Sharma S, Latif Z, Makuvire TT, et al. Readability and accessibility of patient-education materials for heart failure in the United States. J Card Fail. 2025;31(1):154-157. doi:10.1016/j.cardfail.2024.06.015
Ahn AB, Kulhari S, Karimi A, Sundararajan S, Sajatovic M. Readability of patient education material in stroke: a systematic literature review. Top Stroke Rehabil. 2024;31(4):345-360. doi:10.1080/10749357.2023.2259177
Ruan T, Shao X, Sun Y, et al. Evaluation of accuracy, quality, and readability of information on hypothyroidism provided by different Artificial Intelligence chatbot models. Front Public Health. 2025;13: 1698596. doi:10.3389/fpubh.2025.1698596
Büker M, Mercan G. Readability, accuracy, and appropriateness and quality of AI chatbot responses as a patient information source on root canal retreatment: a comparative assessment. Int J Med Inform. 2025; 201:105948. doi:10.1016/j.ijmedinf.2025.105948
Misra AR, Oermann MH, Teague MS, Ledbetter LS. An evaluation of websites offering caregiver education for tracheostomy and home mechanical ventilation. J Pediatr Nurs. 2021;56:64-69. doi:10.1016/j.pedn.2020.09.014
Hanci V, Oner O, Ozduran E, Gokel E. YouTube as a source of information about percutaneous tracheostomy. Gazi Med J. 2023;34:372-378. doi:10.12996/gmj.2023.77
Ozbek IC, Ozduran E, Hanci V. Artificial Intelligence in pain: a comprehensive review. Med J West Black Sea. 2026;10:168-178. doi:10.29058/mjwbs.1881313

Details

Primary Language

English

Subjects

Intensive Care

Journal Section

Research Article

Authors

Ferhan Demirer Aydemir ^*
0000-0002-6740-1496
Türkiye

Volkan Hancı
0000-0002-2227-194X
Türkiye

Publication Date

June 25, 2026

Submission Date

May 12, 2026

Acceptance Date

June 10, 2026

Published in Issue

Year 2026 Volume: 7 Number: 3

IZ

https://izlik.org/JA96YF25AN

Cite

RIS / Bibtex

APA

Demirer Aydemir, F., & Hancı, V. (2026). Readability, reliability, and quality of AI chatbot responses on home mechanical ventilation. Journal of Medicine and Palliative Care, 7(3), 553-561. https://izlik.org/JA96YF25AN

AMA

1.Demirer Aydemir F, Hancı V. Readability, reliability, and quality of AI chatbot responses on home mechanical ventilation. J Med Palliat Care / JOMPAC / jompac. 2026;7(3):553-561. https://izlik.org/JA96YF25AN

Chicago

Demirer Aydemir, Ferhan, and Volkan Hancı. 2026. “Readability, Reliability, and Quality of AI Chatbot Responses on Home Mechanical Ventilation”. Journal of Medicine and Palliative Care 7 (3): 553-61. https://izlik.org/JA96YF25AN.

EndNote

Demirer Aydemir F, Hancı V (June 1, 2026) Readability, reliability, and quality of AI chatbot responses on home mechanical ventilation. Journal of Medicine and Palliative Care 7 3 553–561.

IEEE

[1]F. Demirer Aydemir and V. Hancı, “Readability, reliability, and quality of AI chatbot responses on home mechanical ventilation”, J Med Palliat Care / JOMPAC / jompac, vol. 7, no. 3, pp. 553–561, June 2026, [Online]. Available: https://izlik.org/JA96YF25AN

ISNAD

Demirer Aydemir, Ferhan - Hancı, Volkan. “Readability, Reliability, and Quality of AI Chatbot Responses on Home Mechanical Ventilation”. Journal of Medicine and Palliative Care 7/3 (June 1, 2026): 553-561. https://izlik.org/JA96YF25AN.

JAMA

1.Demirer Aydemir F, Hancı V. Readability, reliability, and quality of AI chatbot responses on home mechanical ventilation. J Med Palliat Care / JOMPAC / jompac. 2026;7:553–561.

MLA

Demirer Aydemir, Ferhan, and Volkan Hancı. “Readability, Reliability, and Quality of AI Chatbot Responses on Home Mechanical Ventilation”. Journal of Medicine and Palliative Care, vol. 7, no. 3, June 2026, pp. 553-61, https://izlik.org/JA96YF25AN.

Vancouver

1.Ferhan Demirer Aydemir, Volkan Hancı. Readability, reliability, and quality of AI chatbot responses on home mechanical ventilation. J Med Palliat Care / JOMPAC / jompac [Internet]. 2026 Jun. 1;7(3):553-61. Available from: https://izlik.org/JA96YF25AN