Research Article

Comparative Evaluation of Claude, Gemini, and ChatGPT in Responding to Patient Inquiries About Hyperprolactinemia and Prolactinoma: A Multi-Rater Cross-Sectional Study Using English-Language Queries

Volume: 5 Number: 1 June 27, 2026
İsmail Engin *, Melda Çelik Tufan , Esma Merve Arda Özkan
EN

Comparative Evaluation of Claude, Gemini, and ChatGPT in Responding to Patient Inquiries About Hyperprolactinemia and Prolactinoma: A Multi-Rater Cross-Sectional Study Using English-Language Queries

Abstract

Objective: This cross-sectional comparative study aimed to compare the accuracy and adequacy of responses generated by three state-of-the-art large language models (LLMs) Claude Opus 4.6, Gemini 3 Pro, and ChatGPT GPT-5.4 to 50 frequently asked English-language questions about hyperprolactinemia and prolactinoma, as evaluated by three independent endocrinologists. Methods: Questions compiled from outpatient practice were presented using a zero-shot, plain-query approach through the publicly available consumer web interfaces. Responses were independently evaluated by three blinded endocrinologists using a 6-point Likert scale for accuracy and a 5-point Likert scale for adequacy. For each question, the three evaluators’ ratings were averaged to a question-level score, producing 50 question-level scores per model. Inter-model comparisons used the Friedman test with Bonferroni-corrected Wilcoxon signed-rank post hoc tests (adjusted α = .017); within-model comparisons across thematic categories used the Kruskal–Wallis test. Results: Claude achieved the highest mean accuracy (5.76 ± 0.64; 95% CI 5.58–5.93) and adequacy (4.52 ± 0.52; 95% CI 4.38–4.67), followed by Gemini (accuracy 5.58 ± 0.73; adequacy 4.31 ± 0.63) and ChatGPT (accuracy 5.39 ± 0.90; adequacy 4.06 ± 0.82). The Friedman test was significant for both metrics (accuracy χ² = 50.54; adequacy χ² = 63.51; both p < .001). All three pairwise comparisons were significant after Bonferroni correction (all p < .001), following the order Claude > Gemini > ChatGPT, with large effect sizes (r = .63 - .72). The pregnancy period category yielded the lowest scores across all models. Conclusion: These findings demonstrate the capacity of LLMs to generate clinically accurate, guideline-concordant English-language content regarding prolactinoma under optimal linguistic conditions. However, because the evaluation was limited to English-language queries, the results cannot be generalized to non-English-speaking populations, and their use as patient education tools requires direct comprehension and readability studies in the target language.

Keywords

Artificial intelligence, Hyperprolactinemia, Large language models, Medical knowledge benchmark, Patient education, Prolactinoma

Ethical Statement

This study did not involve human participants, identifiable patient data, or biological specimens. All questions used for querying the large language models were generated by the authors based on routine clinical practice and were not obtained from any specific patient. Accordingly, Institutional Review Board (IRB) / ethics committee approval was not required.

References

  1. Abbasgholizadeh Rahimi, S., Légaré, F., Sharma, G., Archambault, P., Zomahoun, H. T. V., Chandavong, S., Rheault, N., Wong, T., Langlois, L., Couturier, Y., & Dugas, M. (2021). Application of artificial intelligence in community-based primary health care: Systematic scoping review and critical appraisal. Journal of Medical Internet Research, 23(9), e29839. https://doi.org/10.2196/29839
  2. Aliyeva, A., Nevzati, E., Grassia, F., Riaz, M., Mann, S., & Nasirov, R. (2026). AI at the sella turcica: Multi-model large language model evaluation in pituitary adenomas. Brain & Spine, 6, 105997. https://doi.org/10.1016/j.bas.2026.105997
  3. Auriemma, R. S., Pirchio, R., Pivonello, C., Garifalos, F., Colao, A., & Pivonello, R. (2023). Approach to the patient with prolactinoma. The Journal of Clinical Endocrinology and Metabolism, 108(9), 2400–2423. https://doi.org/10.1210/clinem/dgad174
  4. Baş Aksu, Ö., Aydın, R. F., Gökçay Canpolat, A., Demir, Ö., Şahin, M., Emral, R., & Güllü, S. (2025). Artificial intelligence in endocrine practice: Comparing ChatGPT, Gemini, and Claude for adrenal incidentaloma care. Journal of Endocrinological Investigation, 49(1), 69–79. https://doi.org/10.1007/s40618-025-02715-0
  5. Borque-Fernando, Á., Navarro, D., Doblare, M., Esteban, L. M., Perez-Fentes, D., Álvarez-Maestro, M., & Álvarez-Ossorio Fernández, J. L. (2026). Accuracy of large language models in interpreting urological clinical guidelines: A comparative study with expert evaluation. Therapeutic Advances in Urology, 18, 17562872261436905. https://doi.org/10.1177/17562872261436905
  6. Casanueva, F. F., Molitch, M. E., Schlechte, J. A., Abs, R., Bonert, V., Bronstein, M. D., Brue, T., Cappabianca, P., Colao, A., Fahlbusch, R., Fideleff, H., Hadani, M., Kelly, P., Kleinberg, D., Laws, E., Marek, J., Scanlon, M., Sobrinho, L. G., Wass, J. A. H., & Giustina, A. (2006). Guidelines of the Pituitary Society for the diagnosis and management of prolactinomas. Clinical Endocrinology, 65(2), 265–273. https://doi.org/10.1111/j.1365-2265.2006.02562.x
  7. Dave, T., Athaluri, S. A., & Singh, S. (2023). ChatGPT in medicine: An overview of its applications, advantages, limitations, future prospects, and ethical considerations. Frontiers in Artificial Intelligence, 6, 1169595. https://doi.org/10.3389/frai.2023.1169595
  8. Goodman, R. S., Patrinely, J. R., Stone, C. A., Zimmerman, E., Donald, R. R., Chang, S. S., & Scoville, E. A. (2023). Accuracy and reliability of chatbot responses to physician questions. JAMA Network Open, 6(10), e2336483. https://doi.org/10.1001/jamanetworkopen.2023.36483 Gunesli, I., Aksun, S., Fathelbab, J., & Yildiz, B. O. (2025). Comparative evaluation of ChatGPT-4, ChatGPT-3.5 and Google Gemini on PCOS assessment and management based on recommendations from the 2023 guideline. Endocrine, 88(1), 315–322. https://doi.org/10.1007/s12020-024-04121-7
  9. Khodve, G. B., & Banerjee, S. (2023). Artificial intelligence in efficient diabetes care. Current Diabetes Reviews, 19(8), e050922208561. https://doi.org/10.2174/1573399819666220905163940
  10. Kim, K., Shin, S., Kim, S., & Lee, E. (2023). The relation between eHealth literacy and health-related behaviors: Systematic review and meta-analysis. Journal of Medical Internet Research, 25, e40778. https://doi.org/10.2196/40778
APA
Engin, İ., Çelik Tufan, M., & Arda Özkan, E. M. (2026). Comparative Evaluation of Claude, Gemini, and ChatGPT in Responding to Patient Inquiries About Hyperprolactinemia and Prolactinoma: A Multi-Rater Cross-Sectional Study Using English-Language Queries. Farabi Medical Journal, 5(1). https://doi.org/10.59518/farabimedj.1928053
AMA
1.Engin İ, Çelik Tufan M, Arda Özkan EM. Comparative Evaluation of Claude, Gemini, and ChatGPT in Responding to Patient Inquiries About Hyperprolactinemia and Prolactinoma: A Multi-Rater Cross-Sectional Study Using English-Language Queries. Farabi Medical Journal. 2026;5(1). doi:10.59518/farabimedj.1928053
Chicago
Engin, İsmail, Melda Çelik Tufan, and Esma Merve Arda Özkan. 2026. “Comparative Evaluation of Claude, Gemini, and ChatGPT in Responding to Patient Inquiries About Hyperprolactinemia and Prolactinoma: A Multi-Rater Cross-Sectional Study Using English-Language Queries”. Farabi Medical Journal 5 (1). https://doi.org/10.59518/farabimedj.1928053.
EndNote
Engin İ, Çelik Tufan M, Arda Özkan EM (June 1, 2026) Comparative Evaluation of Claude, Gemini, and ChatGPT in Responding to Patient Inquiries About Hyperprolactinemia and Prolactinoma: A Multi-Rater Cross-Sectional Study Using English-Language Queries. Farabi Medical Journal 5 1
IEEE
[1]İ. Engin, M. Çelik Tufan, and E. M. Arda Özkan, “Comparative Evaluation of Claude, Gemini, and ChatGPT in Responding to Patient Inquiries About Hyperprolactinemia and Prolactinoma: A Multi-Rater Cross-Sectional Study Using English-Language Queries”, Farabi Medical Journal, vol. 5, no. 1, June 2026, doi: 10.59518/farabimedj.1928053.
ISNAD
Engin, İsmail - Çelik Tufan, Melda - Arda Özkan, Esma Merve. “Comparative Evaluation of Claude, Gemini, and ChatGPT in Responding to Patient Inquiries About Hyperprolactinemia and Prolactinoma: A Multi-Rater Cross-Sectional Study Using English-Language Queries”. Farabi Medical Journal 5/1 (June 1, 2026). https://doi.org/10.59518/farabimedj.1928053.
JAMA
1.Engin İ, Çelik Tufan M, Arda Özkan EM. Comparative Evaluation of Claude, Gemini, and ChatGPT in Responding to Patient Inquiries About Hyperprolactinemia and Prolactinoma: A Multi-Rater Cross-Sectional Study Using English-Language Queries. Farabi Medical Journal. 2026;5. doi:10.59518/farabimedj.1928053.
MLA
Engin, İsmail, et al. “Comparative Evaluation of Claude, Gemini, and ChatGPT in Responding to Patient Inquiries About Hyperprolactinemia and Prolactinoma: A Multi-Rater Cross-Sectional Study Using English-Language Queries”. Farabi Medical Journal, vol. 5, no. 1, June 2026, doi:10.59518/farabimedj.1928053.
Vancouver
1.İsmail Engin, Melda Çelik Tufan, Esma Merve Arda Özkan. Comparative Evaluation of Claude, Gemini, and ChatGPT in Responding to Patient Inquiries About Hyperprolactinemia and Prolactinoma: A Multi-Rater Cross-Sectional Study Using English-Language Queries. Farabi Medical Journal. 2026 Jun. 1;5(1). doi:10.59518/farabimedj.1928053