ChatGPT ve Gemini'nin Aşılar ve Bağışıklama Konularındaki Soruları Yanıtlama Performanslarının Karşılaştırılması

Year 2025, Volume: 35 Issue: 5, 1010 - 1018, 28.10.2025

Ayşegül İnci Sezen , Meryem Şahin Özdemir Yusuf Emre Özdemir

https://doi.org/10.54005/geneltip.1735723

Abstract

Giriş/Amaç:
Bu çalışmanın amacı, ChatGPT-4 Plus ve Gemini Advanced sohbet botlarının aşılama ve bağışıklama konularındaki yeterlilik, güvenilirlik ve tekrarlanabilirliklerini değerlendirmek ve karşılaştırmaktır. Yapay zekânın sağlık iletişimindeki artan rolü göz önüne alındığında, bu modellerin performansının değerlendirilmesi, halk sağlığı stratejilerine entegrasyonları açısından önemlidir.
Yöntem:
Dünya Sağlık Örgütü eğitim materyalleri ve ulusal bir yetişkin aşılaması kılavuzundan toplam 56 soru seçildi. Sorular dört başlık altında gruplandırıldı: Mitler ve Yanılgılar, Aşılama ve Bağışıklama, Aşılamanın Temel İlkeleri ve Aşılar Hakkında İddialar ve Gerçekler. İki enfeksiyon hastalıkları uzmanı, sohbet botlarının yanıtlarını 1–5 arası Global Kalite Skoru (GQS) ile bağımsız olarak değerlendirdi. Görüş ayrılığı durumlarında üçüncü bir uzman son kararı verdi. Değerlendirmede gözlemciler arası uyum, tekrarlanabilirlik ve yanıtlar arasındaki korelasyon analiz edildi.
Bulgular:
ChatGPT-4 Plus, yanıtlarının %83,9’unda 5 tam puan alırken, Gemini için bu oran %71,4’tü. Ortalama puanlar benzerdi (4,70±0,72 vs. 4,61±0,75; p=0,116). ChatGPT, “Aşılar Hakkında İddialar” başlığında Gemini’ye göre anlamlı üstünlük gösterdi (p=0,012); diğer başlıklarda anlamlı fark saptanmadı. Her iki model de yüksek düzeyde gözlemciler arası uyum (ChatGPT için κ=0,540, Gemini için κ=0,572) ve orta düzeyde korelasyon (r=0,521, p<0,001) sağladı. Yanıtların tekrarlanabilirlik oranları ChatGPT için %94,6, Gemini için %98,2 olarak bulundu (p=0,332).
Sonuç:
ChatGPT-4 Plus ve Gemini Advanced, aşılama konusunda benzer düzeyde doğru ve kapsamlı bilgi sunmuş, yüksek düzeyde tekrarlanabilirlik ve güvenilirlik göstermiştir. Bu yapay zekâ destekli sohbet robotları, halkı aşılar hakkında bilgilendirme ve sağlık alanındaki yanlış bilgilerin önlenmesinde yararlı araçlar olabilir.

Keywords

Yapay zeka , aşı , bağışıklama , ChatGPT , Gemini , sağlık iletişimi

Supporting Institution

yok

Comparative Evaluation of ChatGPT and Gemini in Answering Questions on Vaccines and Immunization

Abstract

Background/Aims:
This study aimed to evaluate and compare the adequacy, reliability, and reproducibility of ChatGPT-4 Plus and Gemini Advanced chatbots in answering questions about vaccines and immunization. Given the growing role of artificial intelligence in healthcare communication, evaluating the performance of these models is crucial for their effective integration into public health strategies.
Methods:
A total of 56 questions were selected from World Health Organization educational materials and a national guideline on adult vaccination. Questions were categorized into four domains: Myths and Misconceptions, Vaccination and Immunization, Basic Principles of Vaccination, and Claims and Facts about Vaccines. Two infectious disease specialists independently rated chatbot responses using the Global Quality Score (1–5). Discrepancies were resolved by a third evaluator. Inter-rater reliability, reproducibility, and correlation of responses were also assessed.
Results:
ChatGPT-4 Plus received a perfect score (5 points) in 83.9% of responses, while Gemini achieved this in 71.4% of responses. Mean scores were similar (4.70±0.72 vs. 4.61±0.75, p=0.116). ChatGPT-4 Plus outperformed Gemini in the "Claims and Facts" category (p=0.012), while other categories showed no significant differences. Both models demonstrated substantial inter-rater agreement (κ=0.540 for ChatGPT, κ=0.572 for Gemini) and moderate correlation (r=0.521, p<0.001). Reproducibility rates were high: 94.6% for ChatGPT and 98.2% for Gemini (p=0.332).
Conclusions:
ChatGPT-4 Plus and Gemini Advanced provided similarly accurate and comprehensive information on vaccination, with high reproducibility and reliability. These AI chatbots may serve as useful tools for vaccine-related public education and preventing misinformation in healthcare.

Keywords

Artificial intelligence , vaccine , immunization , ChatGPT , Gemini , health communication

Supporting Institution

no funding

References

• 1. Sarantopoulos A, Mastori Kourmpani C, Yokarasa AL, Makamanzi C, Antoniou P, Spernovasilis N, Tsioutis C. Artificial Intelligence in Infectious Disease Clinical Practice: An Overview of Gaps, Opportunities, and Limitations. Trop Med Infect Dis. 2024;9(10):228. 2. Roumeliotis KI, Tselikas ND. ChatGPT and Open AI Models: A Preliminary Review. Future Internet. 2023;15(6):192.
• 3. McIntosh TR, Susnjak T, Liu T, Watters P, Xu D, Liu D, Halgamuge MN. From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape. Technologies. 2025; 13(2):51.
• 4. Tangcharoensathien V, Calleja N, Nguyen T, Purnat TD, D’Agostino M, Garcia Saiso S, et al. Framework for managing the COVID 19 infodemic: methods and results of an online, crowdsourced WHO technical consultation. J Med Internet Res. 2020;22(6):e19659.
• 5. Vaishya R, Misra A, Vaish A. ChatGPT: Is this version good for healthcare and research? Diabetes Metab Syndr. 2023;17(2):102744.
• 6. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198.
• 7. Deiana G, Dettori M, Arghittu A, Azara A, Gabutti G, Castiglia P. Artificial intelligence and public health: evaluating ChatGPT responses to vaccination myths and misconceptions. Vaccines. 2023;11(7):1217.
• 8. Benoit JRA. ChatGPT’s misleading medical advice: An analysis of AI generated responses to patient symptoms. BMJ Health Care Inform. 2023;30(1):e100615.
• 9. Liu HY, Kemmerly T, Valeriano J, Glickman L, Bach PB. Consulting the Digital Doctor: Google Versus ChatGPT as sources of information on breast implant illness and anaplastic large cell lymphoma. Aesthetic Plast Surg. 2024;48(4):590–607.
• 10. Velupillai S, Suominen H, Liakata M, Goeuriot L, Roberts A, Shah H. Evaluating large language models in clinical question answering: Gemini and ChatGPT compared. J Biomed Inform. 2024;145:104429.
• 11. Scott I, Carter S, Coiera E. Clinician checklist for assessing suitability of machine learning applications in healthcare. BMJ Health Care Inform. 2021 Feb;28(1):e100251.
• 12. Yanagita Y, Yokokawa D, Uchida S, Tawara J, Ikusaka M. Accuracy of ChatGPT on Medical Questions in the National Medical Licensing Examination in Japan: Evaluation Study. JMIR Form Res. 2023;7:e48023.
• 13. Laranjo L, Dunn AG, Tong HL, Kocaballi AB, Chen J, Bashir R, Surian D, Gallego B, Magrabi F, Lau AYS, Coiera E. Conversational agents in healthcare: a systematic review. J Am Med Inform Assoc. 2018;25(9):1248-1258.
• 14. Joshi S, Ha E, Amaya A, Mendoza M, Rivera Y, Singh VK. Ensuring Accuracy and Equity in Vaccination Information From ChatGPT and CDC: Mixed-Methods Cross-Language Evaluation. JMIR Form Res. 2024;8:e60939.
• 15. Meyrowitsch DW, Jensen AK, Sørensen JB, Varga TV. AI chatbots and (mis)information in public health: impact on vulnerable communities. Front Public Health. 2023;11:1226776.
• 16. Walker HL, Ghani S, Kuemmerli C,Nebiker CA, Müller BP, Raptis DA, Staubli SM. Reliability of Medical Information Provided by ChatGPT: Assessment Against Clinical Guidelines and Patient Information Quality Instrument. J Med Internet Res 2023;25:e47479.
• 17. Sahin Ozdemir M, Ozdemir YE. Comparison of the performances between ChatGPT and Gemini in answering questions on viral hepatitis. Sci Rep. 2025;15(1):1712.
• 18. Tunçer G, Güçlü KG. How reliable is ChatGPT as a novel consultant in infectious diseases and clinical microbiology? Infect Dis Clin Microbiol. 2024:55–59.
• 19. Liu HY, et al. Consulting the Digital Doctor: Google Versus ChatGPT as sources of information on breast implant associated anaplastic large cell lymphoma and breast Implant illness. Aesthetic Plast Surg. 2024;48(4):590–607.
• 20. Tilton AK, Caplan BE, Cole BJ. Generative AI in consumer health: leveraging large language models for health literacy and clinical safety with a digital health framework. Front Digit Health. 2025;7:1616488.