Assessment and Cross-Language Comparison of ChatGPT’s Responses to Frequently Asked Questions by Pregnant Women: A Multilingual Prospective Observational Study

Abstract

Aim: The objective of this study was to evaluate the efficacy of ChatGPT in providing pregnancy-related health information in English, Turkish, and Arabic, with a focus on medical accuracy, empathetic communication, clarity, and appropriateness of referral to obstetricians. This research aims to address concerns regarding the variability of response accuracy and quality across different languages. Methods: A prospective observational study was conducted within the Department of Obstetrics and Gynecology. Fifteen commonly asked pregnancy-related questions were translated into Turkish and Arabic by certified translators to ensure linguistic accuracy. Responses generated by ChatGPT in English, Turkish, and Arabic were independently assessed by four experienced obstetricians. Evaluation criteria included medical accuracy, empathetic tone, clarity, and appropriateness of referral. Statistical analysis was performed using chi-square tests, with statistical significance defined as p < 0.05. Results: Statistically significant differences were identified across languages in terms of medical accuracy (p = 0.0012), empathetic tone (p < 0.001), and clarity (p < 0.001). Responses provided in English were rated as “excellent” in 73.3% of cases, compared to 40% in Turkish and 41.6% in Arabic. Overall, more than 80% of all responses were rated as either “good” or “excellent.” No significant difference was observed concerning the appropriateness of referral recommendations (p = 0.0885). Conclusions: ChatGPT demonstrates potential as a reliable tool for addressing pregnancy-related health inquiries. However, discrepancies in response accuracy and clarity across languages highlight the need for improved language support, particularly for underrepresented languages. Future research should include a broader range of questions and additional languages to enhance the reliability and clinical applicability of this technology.

Keywords

• Artificial intelligence
• Artificial Neural Networks
• ChatGPT
• Large language model
• Multilingual
• Pregnancy

References

1. 1. Zhang J, Lu JD, Chen B, Pan S, Jin L, Zheng Y, et al. Vision transformer introduces a new vitality to the classification of renal pathology. BMC Nephrol. 2024;25(1):337. https://doi.org/10.1186/s12882-024-03800-x
2. 2. Chen Z, Yadollahpour A. A new era in cognitive neuroscience: the tidal wave of artificial intelligence (AI). BMC Neurosci. 2024;25(1):23. https://doi.org/10.1186/s12868-024-00869-w
3. 3. Raiaan MAK, Mukta MSH, Fatema K, Fahad NM, Sakin S, Mim MMJ. A review on large language models: architectures, applications, taxonomies, open issues and challenges. IEEE Access. 2024;12:26839–26874. https://doi.org/10.1109/ACCESS.2024.3365742
4. 4. Porto BM. Improving triage performance in emergency departments using machine learning and natural language processing: a systematic review. BMC Emerg Med. 2024;24(1):219. https://doi.org/10.1186/s12873-024-01135-2
5. 5. Chung P, Fong CT, Walters AM, Yetisgen M, O’Reilly-Shah VN. Prediction of American Society of Anesthesiologists physical status classification from preoperative clinical text narratives using natural language processing. BMC Anesthesiol. 2023;23(1):296. https://doi.org/10.1186/s12871-023-02248-0
6. 6. Zhao WX, Zhou K, Li J, Tang T, Wang X, Hou Y, et al. A survey of large language models. arXiv:2303.18223 [cs.CL]. https://doi.org/10.48550/arXiv.2303.18223
7. 7. Gulati V, Roy SG, Moawad A, Garcia D, Babu A, Pott JD, et al. Transcending language barriers: can ChatGPT be the key to enhancing multilingual accessibility in healthcare?J Am Coll Radiol 2024;21(12):1888–1895. https://doi.org/10.1016/j.jacr.2024.05.009
8. 8. Grünebaum A, Chervenak J, Pollet SL, Katz A, Chervenak FA. The exciting potential for ChatGPT in obstetrics and gynecology. Am J Obstet Gynecol. 2023;228(6):696–705. https://doi.org/10.1016/j.ajog.2023.03.009

9. 9. Chang Y-H, Lin Y-C, Huang F-W, Chen D-M, Chung Y-T, Chen W-K, et al. Using machine learning and natural language processing in triage for prediction of clinical disposition in the emergency department. BMC Emerg Med. 2024;24:237. https://doi.org/10.1186/s12873-024-01152-1
10. 10. Chiarello F, Giordano V, Spada I, Barandoni S, Fantoni G. Future applications of generative large language models: a data-driven case study on ChatGPT. Technovation. 2024;133:103002. https://doi.org/10.1016/j.technovation.2024.103002
11. 11. Briganti G. How ChatGPT works: a mini review. Eur Arch Otorhinolaryngol. 2024;281(3):1565–1569. https://doi.org/10.1007/s00405-023-08337-7
12. 12. Allen MR, Webb S, Mandvi A, Freiden M, Ming T-S, Kallenberg G. Navigating the doctor-patient-AI relationship-a mixed-methods study of physician attitudes toward artificial intelligence in primary care. BMC Prim Care. 2024;25(1):42.https://doi.org/10.1186/s12875-024-02282-y
13. 13. Wolfram S. WHAT is ChatGPT Doing ... and Why Does It Work? Wolfram Media, Inc.; 2023. 112 p. https://doi.org/10.31855/bc47ee6b-75c
14. 14. Sayakhot P, Carolan-Olah M. Internet use by pregnant women seeking pregnancy-related information: a systematic review. BMC Pregnancy Childbirth. 2016;16:1–10. https://doi. org/10.1186/s12884-016-0856-5
15. 15. Çelik E, Turgut MA, Aydoğan M, Kılınç M, Toktaş İ, Akelma H. Comparison of AI applications and anesthesiologist’s anesthesia method choices. BMC Anesthesiol. 2025;25:2. https://doi. org/10.1186/s12871-024-02882-2
16. 16. Han H, Li R, Fu D, Zhou H, Zhan Z, Wu Y et al. Revolutionizing spinal interventions: a systematic review of artificial intelligence technology applications in contemporary surgery. BMC Surg. 2024;24(1):345. https://doi.org/10.1186/s12893-024-02646-2
17. 17. Serapio A, Chaudhari G, Savage C, Lee YJ, Vella M, Sridhar S, et al. An open-source fine-tuned large language model for radiological impression generation: a multi-reader performance study. BMC Med Imaging. 2024;24(1):254. https://doi.org/10.1186/s12880-024-01435-w
18. 18. Haver HL, Ambinder EB, Bahl M, Oluyemi ET, Jeudy J,nYi PH. Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT. Radiology. 2023;307(4):e230424. https://doi.org/10.1148/radiol.230424
19. 19. Li SW, Kemp MW, Logan SJ, Dimri PS, Singh N, Mattar CNZ, et al. ChatGPT outscored human candidates in a virtual objective structured clinical examination in obstetrics and gynecology. Am J Obstet Gynecol. 2023;229(2):172.e1–72.e12. https://doi.org/10.1016/j.ajog.2023.04.020
20. 20. Hopkins BS, Nguyen VN, Dallas J, Texakalidis P, Yang M, Renn A, et al. ChatGPT versus the neurosurgical written boards: a comparative analysis of artificial intelligence/ machine learning performance on neurosurgical board-style questions. J Neurosurg. 2023;139(3):904–911. https://doi.org/10.3171/2023.2.JNS23419
21. 21. Davies NP, Wilson R, Winder MS, Tunster SJ, McVicar K, Thakrar S, et al. ChatGPT sits the DFPH exam: large language model performance and potential to support public health learning. BMC Med Educ. 2024;24(1):57. https://doi.org/10.1186/s12909-024-05042-9
22. 22. Zong H, Li J, Wu E, Wu R, Lu J, Shen B. Performance of ChatGPT on Chinese national medical licensing examinations:a five-year examination evaluation study for physicians, pharmacists and nurses. BMC Med Educ. 2024;24(1):143. https://doi.org/10.1186/s12909-024-05125-7
23. 23. DeMarco J, Nystrom M. The importance of health literacy in patient education. J Consum Health Internet. 2010;14(3):294–301. https://doi.org/10.1080/15398285.2010.502021
24. 24. Ye Z, Zhang B, Zhang K, Mendez MJG, Yan H, We T, et al. An assessment of ChatGPT’s responses to frequently asked questions about cervical and breast cancer. BMC Womens Health. 2024;24(1):482. https://doi.org/10.1186/s12905-024-03320-8