Yapay Zeka Doğum Uzmanının Yerine Konuşursa: Gerçek Hasta Sorularında Çok Dilli Doğruluk

Öz

Amaç Sağlık alanında üretken büyük dil modellerinin (LLM) kullanımı hızla artmaktadır. Ancak çok dilli doğrulukları ve sundukları bilimsel referansların güvenilirliği hakkında kapsamlı veri sınırlıdır. Bu çalışma, üç ücretsiz LLM’nin (ChatGPT, Google Gemini, DeepSeek) Türkçe ve İngilizce gebelik sorularındaki yanıt doğruluğunu ve referans kalitesini karşılaştırmayı amaçladı. Yöntem Karşılaştırmalı gözlemsel çalışmada, klinikte sık karşılaşılan 14 gebelik sorusu her modele Türkçe ve İngilizce olarak yöneltildi ve güncel bilimsel web kaynaklarıyla yanıt vermeleri istendi. Yanıtlar modeli bilmeden kadın hastalıkları ve doğum uzmanları tarafından doğruluk açısından değerlendirildi. Referanslar ayrı bir ekip tarafından güvenilirlik ve bilimsel geçerlilik açısından incelendi. İstatistiksel analiz yapıldı. Bulgular Dil ve model altyapısının performansı anlamlı şekilde etkilediği görüldü. Google Gemini ve DeepSeek İngilizce sorularda Türkçe’ye göre daha yüksek doğruluk gösterdi (p<0.05). ChatGPT’de dil bazlı fark saptanmadı. İngilizce sorularda Gemini ve DeepSeek, ChatGPT’den daha doğruydu (p<0.05); Türkçe sorularda anlamlı fark yoktu. Referans güvenilirliği açısından ChatGPT her iki dilde de en yüksek “mükemmel referans” oranına sahipti (Türkçe %78.5, İngilizce %71.4). Sonuç LLM’ler gebelik hakkında genel bilgi sunabilir; ancak dil yeterliliği ve referans güvenilirliği açısından önemli sınırlılıkları vardır. Klinik karar gerektiren durumlarda tek başına kullanımları uygun değildir. Ana dil odaklı eğitim, referans doğrulama sistemlerinin güçlendirilmesi ve hekim denetiminin entegrasyonu gereklidir.

Anahtar Kelimeler

• Yapay zeka
• gebelik
• LLM
• ChatGPT
• DeepSeek
• Gemini

Destekleyen Kurum

Hiçbir kurumdan finansal destek alınmamıştır.

Etik Beyan

Bu çalışma, insan katılımcılardan veri toplanmasını içermemekte ve herhangi bir deneysel müdahaleyi kapsamamaktadır; bu nedenle, etik kurul onayı gerekmemektedir.

Teşekkür

yok

When Artificial Intelligence Speaks For The Obstetrician: Multilingual Accuracy On Real Patient Questions

Öz

Background The use of generative large language models (LLMs) in healthcare is rapidly increasing, offering easier access to medical information. However, comprehensive data on their multilingual accuracy and the reliability of cited scientific references remain limited. This study aimed to compare the response accuracy and reference quality of three free LLMs (ChatGPT, Google Gemini, DeepSeek) in Turkish and English using common pregnancy-related questions. Methods In this comparative observational study, 14 frequently asked pregnancy questions were posed to each LLM in Turkish and English, requesting responses supported by up-to-date scientific web sources. Answers were evaluated blindly by obstetricians and gynaecologists for accuracy. References were independently assessed for reliability, scientific validity, and accessibility. Statistical analyses were performed. Results Language and model infrastructure significantly influenced performance. Google Gemini and DeepSeek provided more accurate responses in English than in Turkish (p<0.05), while ChatGPT showed no language-based accuracy difference. In English, Gemini and DeepSeek were more accurate than ChatGPT (p<0.05); in Turkish, no significant difference was observed. ChatGPT demonstrated the highest rate of “perfect references” in both Turkish (78.5%) and English (71.4%). Conclusions LLMs may provide general pregnancy information but have important limitations in language performance and reference reliability. Their standalone use in clinical decision-making is not appropriate. Enhancing native-language training, improving reference validation systems, and integrating physician supervision are essential to optimise AI-supported healthcare.

Anahtar Kelimeler

• Artificial intelligence
• pregnancy
• LLM
• ChatGPT
• DeepSeek
• Gemini

Destekleyen Kurum

No financial support was received from any institution.

Etik Beyan

This study does not involve data collection from human participants and does not include any experimental intervention; therefore, ethical committee approval is not required.

Teşekkür

none

Kaynakça

1. Rajpurkar P, Chen E, Banerjee O, Topol EJ. AI in health and medicine. Nat Med. 2022;28(1):31–8. doi:10.1038/s41591-021-01614-0 PubMed PMID: 35058619.
2. Kulkarni PA, Singh H. Artificial Intelligence in Clinical Diagnosis: Opportunities, Challenges, and Hype. JAMA. 2023; 330(4):317–8. doi:10.1001/jama.2023.11440 PubMed PMID: 37410477.
3. Amarasingham R, Audet AMJ, Bates DW, Glenn Cohen I, Entwistle M, Escobar GJ, et al. Consensus Statement on Electronic Health Predictive Analytics: A Guiding Framework to Address Challenges. EGEMS (Wash DC). 2016;4(1):1163. doi:10.13063/2327-9214.1163 PubMed PMID: 27141516.
4. Dhindsa K, Bhandari M, Sonnadara RR. What’s holding up the big data revolution in healthcare? BMJ. 2018;363:k5357. doi:10.1136/bmj.k5357 PubMed PMID: 30593447. Skryd A, Lawrence K. ChatGPT as a Tool for Medical Education and Clinical Decision-Making on the Wards: Case Study. JMIR Form Res. 2024 May 8;8:e51346. doi:10.2196/51346 PubMed PMID: 38717811.
5. Liu S, Wright AP, Patterson BL, Wanderer JP, Turer RW, Nelson SD, et al. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inform Assoc. 2023; 30(7):1237–45. doi:10.1093/jamia/ocad072 PubMed PMID: 37087108.
6. Ayers JW, Poliak A, Dredze M, Leas EC, Zhu Z, Kelley JB, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern Med. 2023; 183(6):589–96. doi:10.1001/jamainternmed.2023.1838 PubMed PMID: 37115527.
7. Ray PP. ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems. 2023;3:121–54. doi:10.1016/j.iotcps.2023.04.003
8. Nakhleh A, Spitzer S, Shehadeh N. ChatGPT’s Response to the Diabetes Knowledge Questionnaire: Implications for Diabetes Education. Diabetes Technol Ther. 2023; 25(8):571–3. doi:10.1089/dia.2023.0134

9. Yeo YH, Samaan JS, Ng WH, Ting PS, Trivedi H, Vipani A, et al. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol. 2023; 29(3):721–32. doi:10.3350/cmh.2023.0089
10. Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J. 2019;6(2):94–8. doi:10.7861/futurehosp.6-2-94 PubMed PMID: 31363513.
11. Li SW, Kemp MW, Logan SJS, Dimri PS, Singh N, Mattar CNZ, et al. ChatGPT outscored human candidates in a virtual objective structured clinical examination in obstetrics and gynecology. Am J Obstet Gynecol. 2023;229(2):172.e1-172.e12. doi:10.1016/j.ajog.2023.04.020
12. Giakas AM, Narayanan R, Ezeonu T, Dalton J, Lee Y, Henry T, et al. Assessing the accuracy and utility of ChatGPT responses to patient questions regarding posterior lumbar decompression. Artificial Intelligence Surgery. 2024; 4(3):233–46. doi:10.20517/ais.2024.24
13. West M, Alsaidi A, Siddiqi R, Sayyed F, Counts R, Quinto L, et al. Artificial intelligence in obstetrics and gynecology: Evaluating ChatGPT and Google Gemini in answering patient questions. International Journal of Gynecology & Obstetrics. 2026;173(1):328–35. doi:10.1002/ijgo.70622
14. 1Sallam M, Al-Mahzoum K, Alshuaib O, Alhajri H, Alotaibi F, Alkhurainej D, et al. Language discrepancies in the performance of generative artificial intelligence models: an examination of infectious disease queries in English and Arabic. BMC Infect Dis. 2024;24(1):799. doi:10.1186/s12879-024-09725-y
15. Guigue P, Meyer R, Thivolle‐Lioux G, Brezinov Y, Levin G. Performance of ChatGPT in French language Parcours d’Accès Spécifique Santé test and in OBGYN. International Journal of Gynecology & Obstetrics. 2024;164(3):959–63. doi:10.1002/ijgo.15083
16. Morreel S, Mathysen D, Verhoeven V. Aye, AI! ChatGPT passes multiple-choice family medicine exam. Med Teach. 2023 Jul; 45(6):665–6. doi:10.1080/0142159X.2023.2187684
17. Taira K, Itaya T, Hanada A. Performance of the Large Language Model ChatGPT on the National Nurse Examinations in Japan: Evaluation Study. JMIR Nurs. 2023;6:e47305. doi:10.2196/47305
18. Rosoł M, Gąsior JS, Łaba J, Korzeniewski K, Młyńczak M. Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination. Sci Rep. 2023;13(1):20512. doi:10.1038/s41598-023-46995-z
19. Gobira M, Nakayama LF, Moreira R, Andrade E, Regatieri CVS, Belfort Jr. R. Performance of ChatGPT-4 in answering questions from the Brazilian National Examination for Medical Degree Revalidation. Rev Assoc Med Bras. 2023;69(10). doi:10.1590/1806-9282.20230848
20. Wang X, Ye H, Zhang S, Yang M, Wang X. Evaluation of the Performance of Three Large Language Models in Clinical Decision Support: A Comparative Study Based on Actual Cases. J Med Syst. 2025;49(1):23. doi:10.1007/s10916-025-02152-9
21. Giannakopoulos K, Kavadella A, Aaqel Salim A, Stamatopoulos V, Kaklamanos EG. Evaluation of the Performance of Generative AI Large Language Models ChatGPT, Google Bard, and Microsoft Bing Chat in Supporting Evidence-Based Dentistry: Comparative Mixed Methods Study. J Med Internet Res. 2023;25:e51580. doi:10.2196/51580
22. Graf EM, McKinney JA, Dye AB, Lin L, Sanchez-Ramos L. Exploring the Limits of Artificial Intelligence for Referencing Scientific Articles. Am J Perinatol. 2024;41(15):2072–81. doi:10.1055/s-0044-1786033
23. Sebo P. How accurate are the references generated by ChatGPT in internal medicine? Intern Emerg Med. 2024;19(1):247–9. doi:10.1007/s11739-023-03484-5
24. Crossnohere NL, Elsaid M, Paskett J, Bose-Brill S, Bridges JFP. Guidelines for Artificial Intelligence in Medicine: Literature Review and Content Analysis of Frameworks. J Med Internet Res. 2022;24(8):e36823. doi:10.2196/36823
25. Implementing Telehealth in Practice: ACOG Committee Opinion Summary, Number 798. Obstetrics and gynecology. 2020;135(2):493–4. doi:10.1097/AOG.0000000000003672 PubMed PMID: 31977794.