Koroner arter bypass cerrahisi sonrası hasta bilgilendirmesinde güncel sohbet robotlarının performansının değerlendirilmesi

Year 2025, Volume: 8 Issue: 5, 879 - 883, 16.09.2025

Gökhan Yüksel , Selami Gürkan

https://doi.org/10.32322/jhsm.1752483

Abstract

Amaç: Bu çalışma, koroner arter bypass grefti (CABG) ameliyatı sonrası iyileşme süreciyle ilgili hasta bilgilendirmesinde önde gelen yapay zekâ (YZ) sohbet robotlarının kalite, okunabilirlik ve genel performanslarını değerlendirmeyi amaçlamaktadır.
Yöntem: CABG sonrası iyileşme sürecine ilişkin standartlaştırılmış bir tıbbi komut, GPT-4o, GPT 4.1, Grok-4, Claude Opus-4, DeepSeek R1, Gemini Pro, Microsoft Copilot, Llama 4, Mistral Large 2 ve Perplexity Sonar olmak üzere on popüler YZ sohbet robotuna girildi. Yanıtların kalitesi geçerliliği kanıtlanmış iki ölçüm sistemiyle değerlendirildi: modifiye Ensuring Quality Information for Patients (mEQIP) ve Quality Analysis of Medical Artificial Intelligence (QAMAI). Okunabilirlik seviyeleri ise sekiz standart formülü birleştiren Average Reading Level Consensus (ARLC) hesaplayıcısıyla ölçüldü.
Bulgular: Test edilen sohbet robotları arasında, Perplexity Sonar en yüksek mEQIP puanını (%91,7) ve en yüksek QAMAI puanına (30 üzerinden 29) sahipti. En düşük puanlar ise Gemini Pro'ya ait olup, %72,2 mEQIP ve 25/30 QAMAI olarak saptandı. Tüm platformlarda ortalama mEQIP skoru %80,43, ortalama QAMAI skoru ise 27/30 olarak belirlendi ve bu da genelde ortalama olarak yüksek kaliteli yanıtlar verildiğini gösterdi. Okunabilirlik analizine göre, DeepSeek R1 anlaşılması en kolay içeriği üretirken (ARLC: 9,92; 15–16 yaş seviyesi), Llama 4 en karmaşık çıktıyı sundu (ARLC: 14,69; 23+ yaş seviyesi). Tüm sohbet robotlarının ortalama ARLC skoru 11,9 olup, bu düzey üniversite seviyesine karşılık gelmektedir ve hasta eğitimi materyalleri için önerilen 6–8. sınıf seviyesinin oldukça üzerindedir.
Sonuç: YZ sohbet robotları, CABG sonrası hasta bilgilendirmesinde umut verici bir potansiyele sahiptir ve içerik kalitelerinin değerlendirmelerinde sıklıkla yüksek puanlar almışlardır. Ancak okunabilirlik, içerik bütünlüğü ve kaynak şeffaflığı açısından tutarsızlıklar mevcuttur. YZ tarafından üretilen sağlık bilgilerinin giderek daha karmaşık hale gelmesine rağmen, yüksek okuma seviyeleri ve düzensiz atıf uygulamaları, genel hasta kitlesi için erişilebilirliği sınırlayabilir. Gelecekteki sohbet robotları sürümlerinin, hasta merkezli tasarımı, tıbbi kılavuzlara uyumu ve içerik sadeleştirmeyi önceliklendirmesi önerilmektedir.

Keywords

Koroner Arter Bypass Greftleme , Yapay Zekâ , Sohbet Robotları

Ethical Statement

insan verisi kullanılmadığından etik kurul onayına gerek duyulmamıştır.

Evaluation of the performance of current Artificial Intelligence Chatbots regarding patient information after coronary artery bypass surgery

Abstract

Aims: This study aims to evaluate the performance of existing Artificial Intelligence (AI) driven Chatbots regarding patient information after coronary artery bypass (CABG) surgery.
Methods: On July 1, 2025, a standardized medical prompt concerning the recovery process after CABG was submitted to ten prominent AI Chatbots: GPT-4o, GPT 4.1, Grok-4, Claude Opus-4, DeepSeek R1, Gemini Pro, Microsoft Copilot, Llama 4, Mistral Large 2, and Perplexity Sonar. Each response was assessed using two validated scoring systems; the modified Ensuring Quality Information for Patients (mEQIP) and the newly developed Quality Analysis of Medical Artificial Intelligence (QAMAI). Readability was evaluated using the average reading level consensus (ARLC) calculator, which aggregates eight standard readability formulas.
Results: Among the tested Chatbots, Perplexity Sonar achieved the highest mEQIP score (91.7%) and the highest QAMAI score (29/30), while Gemini Pro received the lowest scores in both evaluations (72.2% mEQIP, 25/30 QAMAI). The average mEQIP score across all platforms was 80.43%, and the mean QAMAI score was 27/30, indicating generally high-quality responses. Readability assessment revealed that DeepSeek R1 provided the most comprehensible content (ARLC: 9.92, equivalent to a reading age of 15-16 years), while Llama 4 produced the most complex output (ARLC: 14.69, age 23+). The average ARLC across all Chatbots was 11.9, which corresponds to a college-level reading difficulty and exceeds the recommended sixth to eighth-grade readability level for patient education materials.
Conclusion: AI Chatbots show promising capabilities in delivering post-CABG patient information, often achieving high scores in quality assessments. However, inconsistencies remain in readability, completeness, and source transparency. Despite the increasing sophistication of AI-generated health information, the elevated reading levels and inconsistent citation practices may hinder accessibility for general patient populations. To enhance their role in patient education, future Chatbot iterations should prioritize user-centered design, medical guideline compliance, and content simplification.

Keywords

Coronary artery bypass grafting , Artificial Intelligence , Chatbots

Ethical Statement

Ethical Statement: No ethical approval was needed because this is not a human study, but only online information was used.

References

• Turkey Statistical Agency 2013 Statistical Analysis, 1 April 2014 issue: 16162.
• Bhatnagar P, Wickramasinghe K, Wilkins E, Townsend N. Trends in the epidemiology of cardiovascular disease in the UK. Heart. 2016;102(24): 1945-1952. doi:10.1136/heartjnl-2016-309573
• Melly L, Torregrossa G, Lee T, Jansens JL, Puskas JD. Fifty years of coronary artery bypass grafting. J Thorac Dis. 2018;10(3):1960-1967. doi: 10.21037/jtd.2018.02.43
• Montrief T, Koyfman A, Long B. Coronary artery bypass graft surgery complications: a review for emergency clinicians. Am J Emerg Med. 2018;36(12):2289-2297. doi:10.1016/j.ajem.2018.09.014
• Wai Tai C, Wing Kwan L, Chan J, Angelini GD. Ensuring quality information for patients tool to assess patient information on CABG websites: systemic search and evaluation. Perfusion. 2025;40(6):1468-1476. doi:10.1177/02676591241303842
• Hesse BW, Nelson DE, Kreps GL, et al. Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey. Arch Intern Med. 2005;165(22):2618-2624. doi:10.1001/archinte.165.22.2618
• Silver MP. Patient perspectives on online health information and communication with doctors: a qualitative study of patients 50 years old and over. J Med Internet Res. 2015;17(1):e19. doi:10.2196/jmir.3588
• King MR. The future of AI in medicine: a perspective from a Chatbot. Ann Biomed Eng. 2023;51(2):291-295. doi:10.1007/s10439-022-03121-w
• Şahin MF, Doğan Ç, Topkaç EC, et al. Which current Chatbot is more competent in urological theoretical knowledge? A comparative analysis by the European board of urology in-service assessment. World J Urol. 2025;43(1):116. doi:10.1007/s00345-025-05499-3
• Şahin MF, Topkaç EC, Doğan Ç, et al. Still using only ChatGPT? The comparison of five different Artificial Intelligence Chatbots’ answers to the most common questions about kidney stones. J Endourol. 2024; 38(11):1172-1177. doi:10.1089/end.2024.0474
• Moult B, Franck LS, Brady H. Ensuring quality information for patients: development and preliminary validation of a new instrument to improve the quality of written health care information. Health Expect. 2004;7(2):165-175. doi:10.1111/j.1369-7625.2004.00273.x
• Anıl H, Kayra MV. The digital dialogue on premature ejaculation: evaluating the efficacy of Artificial Intelligence-driven responses. Int Urol Nephrol. 2025;57(9):2829-2836. doi:10.1007/s11255-025-04461-x
• Melloul E, Raptis DA, Oberkofler CE, Dutkowski P, Lesurtel M, Clavien PA. Donor information for living donor liver transplantation: where can comprehensive information be found? Liver Transpl. 2012;18(8):892-900. doi:10.1002/lt.23442
• McCool ME, Wahl J, Schlecht I, Apfelbacher C. Evaluating written patient information for eczema in German: comparing the reliability of two instruments, DISCERN and EQIP. PLoS One. 2015;10(10):e0139895. doi:10.1371/journal.pone.0139895
• Malak A, Şahin MF. How Useful are current Chatbots regarding urology patient information? Comparison of the ten most popular Chatbots’ responses about female urinary incontinence. J Med Syst. 2024;48(1): 102. doi:10.1007/s10916-024-02125-4
• Goodman RS, Patrinely JR, Stone CA Jr, et al. Accuracy and reliability of Chatbot responses to physician questions. JAMA Netw Open. 2023;6(10): e2336483. doi:10.1001/jamanetworkopen.2023.36483
• Wilson EA, Makoul G, Bojarski EA, et al. Comparative analysis of print and multimedia health materials: a review of the literature. Patient Educ Couns. 2012;89(1):7-14. doi:10.1016/j.pec.2012.06.007
• Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011;155(2):97-107. doi:10.7326/0003-4819-155-2-201107 190-00005
• Hua HU, Kaakour AH, Rachitskaya A, Srivastava S, Sharma S, Mammo DA. Evaluation and comparison of ophthalmic scientific abstracts and references by current Artificial Intelligence Chatbots. JAMA Ophthalmol. 2023;141(9):819-824. doi:10.1001/jamaophthalmol.2023.3119