Evaluating Large Language Models for Biomedical Text Summarization: A Study of Cardiovascular Research

Abstract

In this study, presented a comprehensive evaluation of abstractive and extractive summarization performance across three prominent large language models (LLMs): ChatGPT, DeepSeek, and Gemini. A total of 8,000 cardiovascular-related research abstracts were collected from PubMed and summarized using two distinct prompting strategies: abstractive and extractive. This process yielded a dataset of 48,000 summaries. To assess summarization quality, applied a multi-metric evaluation framework including semantic similarity (SBERT cosine), BLEU, GLEU, ROUGE-1 F1, ROUGE-2 F1, ROUGE-L F1, and METEOR. The results indicate that extractive summaries, particularly those generated by ChatGPT, consistently achieve higher scores across most metrics, suggesting stronger lexical fidelity and sequence retention. While Gemini shows balanced performance between abstraction and extraction, DeepSeek yields lower scores in both approaches. This work highlights critical differences in LLM behavior depending on the summarization method and offers a benchmark dataset and evaluation pipeline for future research on AI-assisted biomedical summarization.

Keywords

• Large Language Models (LLMs)
• Abstractive Summarization
• Extractive Summarization
• Cardiovascular Research
• Natural Language Processing(NLP)
• Evaluation Metrics

Supporting Institution

NA

Project Number

NA

Ethical Statement

NA

Thanks

NA

Biyomedikal Metin Özetleme için Büyük Dil Modellerinin Değerlendirilmesi: Kardiyovasküler Araştırmalar Üzerine Bir Çalışma

Abstract

Bu çalışmada, üç önde gelen büyük dil modeli (LLM) — ChatGPT, DeepSeek ve Gemini — üzerinde özetleme performansı kapsamlı biçimde değerlendirilmiştir. PubMed’den toplam 8.000 kardiyovasküler araştırma özeti toplanmış ve bu özetler iki farklı istem stratejisi kullanılarak (abstraktif ve ekstraktif) özetlenmiştir. Bu süreçte 48.000 özetlik bir veri kümesi elde edilmiştir. Özetleme kalitesini değerlendirmek için semantik benzerlik (SBERT cosine), BLEU, GLEU, ROUGE-1 F1, ROUGE-2 F1, ROUGE-L F1 ve METEOR gibi çoklu metriklerden oluşan bir değerlendirme çerçevesi kullanılmıştır. Sonuçlar, özellikle ChatGPT tarafından üretilen ekstraktif özetlerin, çoğu metrikte daha yüksek skorlar elde ederek daha güçlü sözcüksel sadakat ve dizi bütünlüğü sağladığını göstermektedir. Gemini, abstraktif ve ekstraktif yöntemler arasında dengeli bir performans sergilerken, DeepSeek her iki yaklaşımda da daha düşük skorlar vermektedir. Bu çalışma, özetleme yöntemine bağlı olarak LLM davranışındaki kritik farklılıkları vurgulamakta ve gelecekte yapay zekâ destekli biyomedikal özetleme araştırmaları için bir kıyas veri kümesi ile değerlendirme altyapısı sunmaktadır.

Keywords

• Büyük Dil Modelleri
• Abstraktif Özetleme
• Ekstraktif Özetleme
• Kardiyovasküler Araştırma
• Doğal Dil İşleme
• Değerlendirme Ölçütleri

Supporting Institution

Yok

Project Number

NA

Ethical Statement

Yok

Thanks

Yok

References

1. U.S. National Library of Medicine, "Search results for cardiovascular diseases," [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/?term=cardiovascular+diseases. [Accessed: 27-Jun-2025].
2. Z. Lu, "PubMed and beyond: a survey of web tools for searching biomedical literature," Database (Oxford), 2011, Art. no. baq036, doi: 10.1093/database/baq036.
3. B. C. Wallace, J. Kuiper, A. Sharma, M. B. Zhu, and I. J. Marshall, "Extracting PICO sentences from clinical trial reports using supervised distant supervision," Journal of Machine Learning Research, vol. 17, Art. no. 132, 2016.
4. A. Rahman et al., "Comparative analysis based on DeepSeek, ChatGPT, and Google Gemini: Features, techniques, performance, future prospects," arXiv preprint, arXiv:2503.04783, 2025.
5. H. Zhang, X. Liu, and J. Zhang, "Extractive summarization via ChatGPT for faithful summary generation," Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, Dec. 6–10, 2023.
6. M. Wang, M. Wang, F. Yu, Y. Yang, J. Walker, and J. Mostafa, "A systematic review of automatic text summarization for biomedical literature and EHRs," Journal of the American Medical Informatics Association, vol. 28, no. 10, pp. 2287–2297, 2021, doi: 10.1093/jamia/ocab143.
7. A. Chaves, C. Kesiku, and B. Garcia-Zapirain, "Automatic text summarization of biomedical text data: A systematic review," Information, vol. 13, no. 8, Art. no. 393, 2022, doi: 10.3390/info13080393.
8. N. Kemaloğlu Alagöz and E. U. Küçüksille, "System of automatic scientific article summarization in Turkish," Pamukkale University Journal of Engineering Sciences, vol. 30, no. 4, pp. 470–481, 2024.

9. G. Quer and E. J. Topol, "The potential for large language models to transform cardiovascular medicine," The Lancet Digital Health, vol. 6, no. 10, pp. e767–e771, 2024.
10. Q. Xie, Z. Luo, B. Wang, and S. Ananiadou, "A survey for biomedical text summarization: From pre-trained to large language models," arXiv preprint, arXiv:2304.08763, 2023.
11. A. Givchi, R. Ramezani, and A. Baraani-Dastjerdi, "Graph-based abstractive biomedical text summarization," Journal of Biomedical Informatics, vol. 132, Art. no. 104099, 2022.
12. L. Bednarczyk et al., "Scientific evidence for clinical text summarization using large language models: Scoping review," Journal of Medical Internet Research, vol. 27, Art. no. e68998, 2025, doi: 10.2196/68998.
13. L. Tang et al., "Evaluating large language models on medical evidence summarization," NPJ Digital Medicine, vol. 6, no. 1, Art. no. 158, 2023.
14. E. Croxford et al., "Evaluation of large language models for summarization tasks in the medical domain: A narrative review," arXiv preprint, arXiv:2409.18170, 2024.
15. D. Van Veen et al., "Adapted large language models can outperform medical experts in clinical text summarization," Nature Medicine, vol. 30, pp. 1134–1142, 2024, doi: 10.1038/s41591-024-02855-5.
16. U.S. National Library of Medicine, "PubMed overview," [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/about/. [Accessed: 30-Jun-2025].
17. OpenAI, "GPT-4 technical report," [Online]. Available: https://openai.com/research/gpt-4. [Accessed: 30-Jun-2025].
18. S. Mercer, S. Spillard, and D. P. Martin, "Brief analysis of DeepSeek R1 and its implications for generative AI," arXiv preprint, arXiv:2502.02523, 2025.
19. M. E. de Carvalho Souza and L. Weigang, "Grok, Gemini, ChatGPT and DeepSeek: Comparison and applications in conversational artificial intelligence," Inteligencia Artificial, vol. 2, no. 1, 2025.
20. D. Jurafsky and J. H. Martin, Speech and Language Processing, 3rd ed. Stanford, CA, USA: Stanford University, 2021.
21. S. Bird, E. Klein, and E. Loper, Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. Sebastopol, CA, USA: O'Reilly Media, Inc., 2009.
22. S. Kannan, V. Gurusamy, S. Vijayarani, J. Ilamathi, and M. Nithya, "Preprocessing techniques for text mining," International Journal of Computer Science & Communication Networks, vol. 5, no. 1, pp. 7–16, 2014.
23. N. Reimers and I. Gurevych, "Sentence-BERT: Sentence embeddings using siamese BERT-networks," arXiv preprint, arXiv:1908.10084, 2019.
24. N. Reimers and I. Gurevych, "Making monolingual sentence embeddings multilingual using knowledge distillation," arXiv preprint, arXiv:2004.09813, 2020.
25. K. Papineni, S. Roukos, T. Ward, and W. J. Zhu, "BLEU: A method for automatic evaluation of machine translation," in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 311–318, Jul. 2002.
26. Y. Wu et al., "Google's neural machine translation system: Bridging the gap between human and machine translation," arXiv preprint, arXiv:1609.08144, 2016.
27. C. Y. Lin, "ROUGE: A package for automatic evaluation of summaries," in Text Summarization Branches Out Workshop, pp. 74–81, Jul. 2004.
28. S. Banerjee and A. Lavie, "METEOR: An automatic metric for MT evaluation with improved correlation with human judgments," in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72, Jun. 2005.