Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts

Burcu Baştürk; Aytuğ Onan

doi:10.70030/sjmakeu.1730246

Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts

Abstract

Heart disease is one of the leading causes of death worldwide, making early detection and diagnosis essential for effective treatment. With advancements in machine learning (ML) and artificial intelligence (AI), these technologies are being increasingly applied in the medical field, particularly for detecting and predicting heart disease. As AI systems become more complex, it becomes important to distinguish between abstracts generated by AI algorithms and those prepared by human experts. This study aims to develop and assess ML approaches to distinguish between human-written and AI-generated (ChatGPT and NLTK) heart disease abstracts. Using a dataset of 15,000 abstracts (5,000 written by humans, 5,000 reworded by ChatGPT, and 5,000 generated using NLTK), various Natural Language Processing (NLP) techniques, such as tokenization, stop word removal, stemming and lemmatization were applied. The text data was transformed into numerical form using TF-IDF vectorization. Different ML models, including K-nearest neighbors (KNN), support vector machines (SVMs), logistic regression, random forest, decision tree were trained and tested for their classification accuracy. This study highlights the significant potential of ML techniques in ensuring transparency and reliability in AI-driven medical decision-making, especially in the area of heart disease diagnosis.

Keywords

References

Internet: World diseases(CVDs), Health Organization, Cardiovascular https://www.who.int/news-room/fact sheets/detail/cardiovascular-diseases-(cvds), 22.09.2024.
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep Learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539
Bhatt, A. (2020). Healthcare predictive analytics using machine learning and deep learning techniques: a survey. Journal of Electrical Systems and Information Technology, 7(2), 13–19.
Russell, S., & Norvig, P. (2010). Artificial Intelligence: A Modern Approach (3rd ed.). Prentice Hall, Upper Saddle River, NJ.
Mitchell, T. M. (1997). Machine Learning. McGraw-Hill, New York.
Jurafsky, D., & Martin, J. H. (2021). Speech and Language Processing (3rd ed.). Pearson, San Francisco, CA.
Krittanawong, C., Zhang, H., Wang, Z., et al. (2017). Artificial Intelligence in Precision Cardiovascular Medicine. Journal of the American College of Cardiology, 69(21), 2657–2664. https://doi.org/10.1016/j.jacc.2017.03.571
Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, P., Heidenreich, P. A., Harrington, R. A., Liang, D. H., Ashley, E. A., & Zou, J. Y. (2020). Video-based AI for beat-to-beat assessment of cardiac function. Nature, 580(7802), 252–256. https://doi.org/10.1038/s41586-020-2145-8

Goodfellow, I., Shlens, J., & Szegedy, C. (2015). Explaining and Harnessing Adversarial Examples. International Conference on Learning Representations (ICLR), 1–9.
Cingillioğlu, İ. (2023). Detecting AI-generated essays: the ChatGPT challenge. International Journal of Information and Learning Technology, 40(3), 259–268. https://doi.org/10.1108/IJILT-03-2023-0043
Ribeiro, M., Singh, S., & Guestrin, C. (2016). Why Should I Trust You? Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16), 1135–1144. https://doi.org/10.1145/2939672.2939778
Sebastiani, A. (2002). Machine Learning in Automated Text Categorization. ACM Computing Surveys, 34(1), 1–47. https://doi.org/10.1145/505282.505283
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language models are few-shot learners (arXiv:2005.14165). Retrieved from https://arxiv.org/abs/2005.14165
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space (arXiv:1301.3781). Retrieved from https://arxiv.org/abs/1301.3781
Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In Nédellec, C., & Rouveirol, C. (eds.), Machine Learning: ECML-98 (Vol. 1398, pp. 137–142). Springer, Berlin, Heidelberg.
Mathur, P., Srivastava, M., Kulshreshtha, A. D., Gulati, P., & Singh, A. (2020). Artificial intelligence, machine learning, and cardiovascular disease. Clinical Medicine Insights: Cardiology, 14, 1–7. https://doi.org/10.1177/1179546820927404.
Gadde, S. S., & Kalli, V. D. R. (2020). Applications of artificial intelligence in medical devices and healthcare. International Journal of Computer Science Trends and Technology, 8, 182–188.
Mishra, V., Sarraju, A., Kalwani, N. M., & Dexter, J. P. (2024). Evaluation of prompts to simplify cardiovascular disease information generated using a large language model: Cross-sectional study. Journal of Medical Internet Research, 26. https://doi.org/10.2196/51795
Bhattaru, A., Yanamala, N., & Sengupta, P. (2024). Revolutionizing cardiology with words: Unveiling the impact of large language models in medical science writing. Canadian Journal of Cardiology.
Liao, W., Liu, Z., Dai, H., Xu, S., Wu, Z., Zhang, Y., Huang, X., Zhu, D., Cai, H., Li, Q., Liu, T., & Li, X. (2023). Differentiating ChatGPT-generated and human-written medical texts: Quantitative study. JMIR Medical Education, 9(1), e48904. https://doi.org/10.2196/48904
Theocharopoulos, P. C., Anagnostou, P., Tsoukala, A., Georgakopoulos, S. V., Tasoulis, S. K., & Plagianakos, V. P. (2023). Detection of fake generated scientific abstracts (arXiv preprint No. 2304.06148). Retrieved from https://arxiv.org/abs/2304.06148
Kumar, V., Bharti, A., Verma, D., & Bhatnagar, V. (2023). Deep dive into language traits of AI-generated abstracts (arXiv preprint No. 2312.10617). Retrieved from https://arxiv.org/abs/2312.10617
Doru, B., Maier, C., Busse, J., Lücke, T., Schönhoff, J., Enax-Krumova, E., Hessler, S., Berger, M., & Tokic, M. (2025). Detecting artificial intelligence–generated versus human-written medical student essays: Semirandomized controlled study. JMIR Medical Education, 11, e62779. https://doi.org/10.2196/62779
Mallapaty, S. (2025). Signs of AI-generated text found in 14% of biomedical abstracts. Nature. https://www.nature.com/articles/d41586-025-02097-6
U.S. National Library of Medicine. PubMed, from https://pubmed.ncbi.nlm.nih.gov, accessed on 2024-09-26.
Biswas, S. S. (2023). Role of Chat GPT in public health. Annals of Biomedical Engineering, 51(5), 868–869. https://doi.org/10.1007/s10439-023-03172-7
Miller, G. A. (1995). WordNet: A lexical database for English. Communications of the ACM, 38, 39–41. https://doi.org/10.1145/219717.219748
Grefenstette, G. (1999). Tokenization. In Voutilainen, A., Heikkilä, J., & Anttila, A. (eds.), Syntactic wordclass tagging, Springer, Dordrecht, pp. 117–133.
Kaur, J., & Buttar, P. K. (2018). A systematic review on stopword removal algorithms. International Journal on Future Revolution in Computer Science & Communication Engineering, 4(4), 207–210.
Kannan, S., & Gurusamy, V. (2014). Preprocessing techniques for text mining. International Journal of Computer Science & Communication Networks, 5(1), 7–16.
Sparck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1), 11–21. https://doi.org/10.1108/eb026526
Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI), Montreal, Canada, pp. 1137–1145.
Hosmer Jr, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression. John Wiley & Sons, New York.
Charbuty, B., & Abdulazeez, A. (2021). Classification based on decision tree algorithm for machine learning. Journal of Applied Science and Technology Trends, 2(1), 20–28. https://doi.org/10.38094/jastt20165
Breiman, L. (2017). Classification and regression trees. Routledge, New York.
Kulkarni, V. Y., & Sinha, P. K. (2012). Pruning of random forest classifiers: A survey and future directions. Proceedings of the International Conference on Data Science & Engineering (ICDSE), pp. 64–68. IEEE.
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018
Abdel Hady, M. F., & Schwenker, F. (2010). Combining committee-based semi-supervised learning and active learning. Journal of Computer Science and Technology, 25(4), 681–698. https://doi.org/10.1007/s11390-010-9357-6 dl.acm.org+8
Han, J., Kamber, M., & Pei, J. (2011). Data Mining: Concepts and Techniques (3rd ed.). Morgan Kaufmann, San Francisco. https://doi.org/10.1016/C2009-0-61819-5
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer, New York.
Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. Cambridge University Press, Cambridge.
Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30.

Details

Primary Language

English

Subjects

Natural Language Processing

Journal Section

Research Article

Authors

Burcu Baştürk
0009-0005-4781-353X
Türkiye

Aytuğ Onan ^*
0000-0002-9434-5880
Türkiye

Early Pub Date

September 1, 2025

Publication Date

December 26, 2025

Submission Date

June 30, 2025

Acceptance Date

August 11, 2025

Published in Issue

Year 2025 Volume: 8 Number: 2

DOI

https://doi.org/10.70030/sjmakeu.1730246

IZ

https://izlik.org/JA78MM75KR

Cite

RIS / Bibtex

APA

Baştürk, B., & Onan, A. (2025). Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts. Scientific Journal of Mehmet Akif Ersoy University, 8(2), 57-70. https://doi.org/10.70030/sjmakeu.1730246

AMA

1.Baştürk B, Onan A. Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts. Techno-Science. 2025;8(2):57-70. doi:10.70030/sjmakeu.1730246

Chicago

Baştürk, Burcu, and Aytuğ Onan. 2025. “Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts”. Scientific Journal of Mehmet Akif Ersoy University 8 (2): 57-70. https://doi.org/10.70030/sjmakeu.1730246.

EndNote

Baştürk B, Onan A (December 1, 2025) Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts. Scientific Journal of Mehmet Akif Ersoy University 8 2 57–70.

IEEE

[1]B. Baştürk and A. Onan, “Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts”, Techno-Science, vol. 8, no. 2, pp. 57–70, Dec. 2025, doi: 10.70030/sjmakeu.1730246.

ISNAD

Baştürk, Burcu - Onan, Aytuğ. “Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts”. Scientific Journal of Mehmet Akif Ersoy University 8/2 (December 1, 2025): 57-70. https://doi.org/10.70030/sjmakeu.1730246.

JAMA

1.Baştürk B, Onan A. Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts. Techno-Science. 2025;8:57–70.

MLA

Baştürk, Burcu, and Aytuğ Onan. “Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts”. Scientific Journal of Mehmet Akif Ersoy University, vol. 8, no. 2, Dec. 2025, pp. 57-70, doi:10.70030/sjmakeu.1730246.

Vancouver

1.Burcu Baştürk, Aytuğ Onan. Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts. Techno-Science. 2025 Dec. 1;8(2):57-70. doi:10.70030/sjmakeu.1730246