Böbrek Nakli İle İlgili Soruları Cevaplamada Chat Generative Pretrained Transformer (ChatGPT) Performansının Değerlendirilmesi

Year 2025, Volume: 20 Issue: 1, 21 - 31, 27.02.2025

Yunus Çolakoğlu , Ali Ayten , Caglar Sertkaya , Kagan Toksal , Serdar Karadağ

https://doi.org/10.33719/nju1613084

Abstract

Giriş
Son yıllarda popüler hale gelen sosyal medya (Youtube, Facebook, Instagram, Twitter vb.) ve yapay zeka (AI) uygulamaları günümüzde hastaların başvurdukları ilk kaynaklardır. ChatGPT, OpenAI tarafından geliştirilen bir yapay zeka destekli dil modelidir ve sağlık sorunları üzerinde başarısı birçok çalışma tarafından gösterilmiştir. Biz bu çalışmamızda ChatGPT'nin böbrek nakli ile ilgili sorulara verdiği yanıtların yeterliliğini değerlendirmeyi amaçladık.
Gereç ve Yöntemler
Hastaların sağlık forumlarında, web sitelerinde ve sosyal medyada (YouTube, Instagram, Twitter) böbrek nakli hakkında sıkça sorduğu sorular analiz edildi. Ayrıca, 2024 Avrupa Üroloji Derneği (EAU) kılavuzunun Böbrek Nakli bölümünün öneri tabloları analiz edildi. Öneri düzeyi güçlü olanlar soru formuna çevrildi. Sorular ChatGPT 4o sürümüne soruldu ve yanıtlar böbrek nakli konusunda deneyimli 2 ürolog ve 1 nefrolog tarafından değerlendirildi.
Bulgular
Değerlendirilen 126 sorudan dışlama kriterleri sonrasında 65 soru ile devam edildi. Cevapların 57’si (%87,6) doğru ve yeterli idi. EAU Kılavuz önerilerine göre 77 adet soru hazırlandı . Soruların 64’ü (%83,1) tamamen doğru cevaplandı. Hem sık sorulan sorularda hem de EAU Kılavuzu’ndan uyarlanan sorularda tamamen yanlış cevap izlenmedi. Soruların tekrarlanabilirliği %100 idi.
Sonuç
Çalışmamız ChatGPT’nin böbrek nakli konusunda güvenilir bir kaynak olduğunu doğrulamaktadır. Gelecekte hem hasta ve yakınlarının hem de sağlık profesyonellerinin sıklıkla başvurabileceği bir platform olacağını düşünmekteyiz.

Keywords

Böbrek nakli, artificial intelligence, ChatGPT

Evaluation of Chat Generative Pretrained Transformer (ChatGPT) Performance in Answering Kidney Transplant Related Questions

Abstract

Objective: Social media such as (Youtube, Facebook, Instagram, Twitter, etc.) and artificial intelligence (AI) are applications that have become popular in recent years, they are the first resources that patients turn to today. ChatGPT is an AI-powered language model developed by OpenAI and its success on health problems are demonstrated by many studies. In this study, we aimed to evaluate the adequacy of ChatGPT’s answers to questions about kidney transplantation.
Material and Methods: Frequently asked questions about kidney transplantation by patients on health forums, websites and social media (YouTube, Instagram, Twitter) were analyzed. We also analyzed the recommendation tables of the Kidney Transplantation section of the 2024 European Association of Urology (EAU) guidelines. Those with strong recommendations were translated into a question form. ChatGPT version 4o questions were asked and the answers were evaluated by 3 urologists experienced in kidney transplantation.
Results: Of the 126 questions evaluated, 65 questions were continued after the exclusion criteria. 57 (87.6%) of the answers were correct and adequate. According to EAU Guideline recommendations, 77 questions were prepared. 64 (83.1%) of the questions were answered completely correctly. There were no completely wrong answers in both frequently asked questions and questions adapted from the EAU Guidelines. Reproducibility of the questions was 100%.
Conclusion: Our study confirms that ChatGPT is a reliable source for kidney transplantation. We think that it will be a platform that both patients and their relatives and healthcare professionals can frequently refer to in the future.

Keywords

kidney transplantation, artificial intelligence, ChatGPT

Ethical Statement

Since no patient data was used in our study, ethics committee approval was not required. Informed Consent: Since no patient data was used in our study, informed consent was not required.

Supporting Institution

The authors declared that this study has received no financial support.

References

• 1. Selen, T., & Merhametsiz, O. (2024). YouTubeTM as a source of information on autosomal dominant polycystic kidney disease: A quality analysis. Digital health, 10, 20552076241248109. https://doi.org/10.1177/20552076241248109
• 2. Zyoud SH, Sweileh WM, Awang R, Al-Jabi SW. Global trends in research related to social media in psychology: Mapping and bibliometric analysis. Int J Ment Health Syst [Internet]. 2018;12(1):1-8. Available from: https://doi.org/10.1186/s13033-018-0182-6
• 3. Zúñiga Salazar, G., Zúñiga, D., Vindel, C. L., et al. (2023). Efficacy of AI Chats to Determine an Emergency: A Comparison Between OpenAI’s ChatGPT, Google Bard, and Microsoft Bing AI Chat. Cureus, 15(9), e45473. https://doi.org/10.7759/cureus.45473
• 4. Caglar, U., Yildiz, O., Meric, A., et al. (2023). Evaluating the performance of ChatGPT in answering questions related to benign prostate hyperplasia and prostate cancer. Minerva urology and nephrology, 75(6), 729-733. https://doi.org/10.23736/S2724-6051.23.05450-2
• 5. Caglar, U., Yildiz, O., Ozervarli, M. et al. (2023). Assessing the Performance of Chat Generative Pretrained Transformer (ChatGPT) in Answering AndrologyRelated Questions. Urology research & practice, 49(6), 365-369. https://doi.org/10.5152/tud.2023.23171
• 6. Secinaro S, Calandra D, Secinaro A, Muthurangu V, Biancone P. The role of artificial intelligence in healthcare: a structured literature review. BMC Med Inform Decis Mak [Internet]. 2021;21(1):1–23. Available from: https://doi.org/10.1186/s12911-021-01488-9
• 7. Faba OR, Boissier R, Budde K, et al. European Association of Urology Guidelines on Renal Transplantation: Update 2024. Eur Urol Focus.
• 8. Stagg BC, Gupta D, Ehrlich JR, et al. HHS Public Access. 2022;4(1):71-7.
• 9. Dubin, J. M., Aguiar, J. A., Lin, J. S., et al. (2024). The broad reach and inaccuracy of men’s health information on social media: analysis of TikTok and Instagram. International journal of impotence research, 36(3), 25-6260. https://doi.org/10.1038/s41443- 022-00645-6
• 10. Samaan JS, Yeo YH, Rajeev N, et al. Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric Surgery. Obes Surg [Internet]. 2023;33(6):1790–6. Available from: https://doi.org/10.1007/s11695-023-06603-5
• 11. Antaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings. Ophthalmol Sci [Internet]. 2023;3(4):100324. Available from: https://doi. org/10.1016/j.xops.2023.100324
• 12. Mankowski, M. A., Jaffe, I. S., Xu, J., et al. (2024). ChatGPT Solving Complex Kidney Transplant Cases: A Comparative Study With Human Respondents. Clinical transplantation, 38(10), e15466. https://doi.org/10.1111/ctr.15466
• 13. Kung, T. H., Cheatham, M., Medenilla, A., et al. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS digital health, 2(2), e0000198. https://doi. org/10.1371/journal.pdig.0000198
• 14. Liu, M., Okuhara, T., Chang, X., et al. (2024). Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis. Journal of medical Internet research, 26, e60807. https://doi.org/10.2196/60807
• 15. Yeo, Y. H., Samaan, J. S., Ng, W. H., et al. (2023). Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clinical and molecular hepatology, 29(3), 721-732. https://doi.org/10.3350/cmh.2023.0089