Machine Learning Based Identification of LLM Generated Scientific Research Article Abstracts

Year 2025, Volume: 8 Issue: 2, 57 - 70

Burcu Baştürk , Aytuğ Onan

https://doi.org/10.70030/sjmakeu.1730246

Abstract

Heart disease is one of the leading causes of death worldwide, making early detection and diagnosis essential for effective treatment. With advancements in machine learning (ML) and artificial intelligence (AI), these technologies are being increasingly applied in the medical field, particularly for detecting and predicting heart disease. As AI systems become more complex, it becomes important to distinguish between abstracts generated by AI algorithms and those prepared by human experts. This study aims to develop and assess ML approaches to distinguish between human-written and AI-generated (ChatGPT and NLTK) heart disease abstracts. Using a dataset of 15,000 abstracts (5,000 written by humans, 5,000 reworded by ChatGPT, and 5,000 generated using NLTK), various Natural Language Processing (NLP) techniques, such as tokenization, stop word removal, stemming and lemmatization were applied. The text data was transformed into numerical form using TF-IDF vectorization. Different ML models, including K-nearest neighbors (KNN), support vector machines (SVMs), logistic regression, random forest, decision tree were trained and tested for their classification accuracy. This study highlights the significant potential of ML techniques in ensuring transparency and reliability in AI-driven medical decision-making, especially in the area of heart disease diagnosis.

Keywords

Machine Learning , Artificial Intelligence , Natural Language Processing , Text Classification

References

• Internet: World diseases(CVDs), Health Organization, Cardiovascular https://www.who.int/news-room/fact sheets/detail/cardiovascular-diseases-(cvds), 22.09.2024.
• LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep Learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539
• Bhatt, A. (2020). Healthcare predictive analytics using machine learning and deep learning techniques: a survey. Journal of Electrical Systems and Information Technology, 7(2), 13–19.
• Russell, S., & Norvig, P. (2010). Artificial Intelligence: A Modern Approach (3rd ed.). Prentice Hall, Upper Saddle River, NJ.
• Mitchell, T. M. (1997). Machine Learning. McGraw-Hill, New York.
• Jurafsky, D., & Martin, J. H. (2021). Speech and Language Processing (3rd ed.). Pearson, San Francisco, CA.
• Krittanawong, C., Zhang, H., Wang, Z., et al. (2017). Artificial Intelligence in Precision Cardiovascular Medicine. Journal of the American College of Cardiology, 69(21), 2657–2664. https://doi.org/10.1016/j.jacc.2017.03.571
• Ouyang, D., He, B., Ghorbani, A., Yuan, N., Ebinger, J., Langlotz, P., Heidenreich, P. A., Harrington, R. A., Liang, D. H., Ashley, E. A., & Zou, J. Y. (2020). Video-based AI for beat-to-beat assessment of cardiac function. Nature, 580(7802), 252–256. https://doi.org/10.1038/s41586-020-2145-8
• Goodfellow, I., Shlens, J., & Szegedy, C. (2015). Explaining and Harnessing Adversarial Examples. International Conference on Learning Representations (ICLR), 1–9.
• Cingillioğlu, İ. (2023). Detecting AI-generated essays: the ChatGPT challenge. International Journal of Information and Learning Technology, 40(3), 259–268. https://doi.org/10.1108/IJILT-03-2023-0043
• Ribeiro, M., Singh, S., & Guestrin, C. (2016). Why Should I Trust You? Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16), 1135–1144. https://doi.org/10.1145/2939672.2939778
• Sebastiani, A. (2002). Machine Learning in Automated Text Categorization. ACM Computing Surveys, 34(1), 1–47. https://doi.org/10.1145/505282.505283
• Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language models are few-shot learners (arXiv:2005.14165). Retrieved from https://arxiv.org/abs/2005.14165
• Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space (arXiv:1301.3781). Retrieved from https://arxiv.org/abs/1301.3781
• Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In Nédellec, C., & Rouveirol, C. (eds.), Machine Learning: ECML-98 (Vol. 1398, pp. 137–142). Springer, Berlin, Heidelberg.
• Mathur, P., Srivastava, M., Kulshreshtha, A. D., Gulati, P., & Singh, A. (2020). Artificial intelligence, machine learning, and cardiovascular disease. Clinical Medicine Insights: Cardiology, 14, 1–7. https://doi.org/10.1177/1179546820927404.
• Gadde, S. S., & Kalli, V. D. R. (2020). Applications of artificial intelligence in medical devices and healthcare. International Journal of Computer Science Trends and Technology, 8, 182–188.
• Mishra, V., Sarraju, A., Kalwani, N. M., & Dexter, J. P. (2024). Evaluation of prompts to simplify cardiovascular disease information generated using a large language model: Cross-sectional study. Journal of Medical Internet Research, 26. https://doi.org/10.2196/51795
• Bhattaru, A., Yanamala, N., & Sengupta, P. (2024). Revolutionizing cardiology with words: Unveiling the impact of large language models in medical science writing. Canadian Journal of Cardiology.
• Liao, W., Liu, Z., Dai, H., Xu, S., Wu, Z., Zhang, Y., Huang, X., Zhu, D., Cai, H., Li, Q., Liu, T., & Li, X. (2023). Differentiating ChatGPT-generated and human-written medical texts: Quantitative study. JMIR Medical Education, 9(1), e48904. https://doi.org/10.2196/48904
• Theocharopoulos, P. C., Anagnostou, P., Tsoukala, A., Georgakopoulos, S. V., Tasoulis, S. K., & Plagianakos, V. P. (2023). Detection of fake generated scientific abstracts (arXiv preprint No. 2304.06148). Retrieved from https://arxiv.org/abs/2304.06148
• Kumar, V., Bharti, A., Verma, D., & Bhatnagar, V. (2023). Deep dive into language traits of AI-generated abstracts (arXiv preprint No. 2312.10617). Retrieved from https://arxiv.org/abs/2312.10617
• Doru, B., Maier, C., Busse, J., Lücke, T., Schönhoff, J., Enax-Krumova, E., Hessler, S., Berger, M., & Tokic, M. (2025). Detecting artificial intelligence–generated versus human-written medical student essays: Semirandomized controlled study. JMIR Medical Education, 11, e62779. https://doi.org/10.2196/62779
• Mallapaty, S. (2025). Signs of AI-generated text found in 14% of biomedical abstracts. Nature. https://www.nature.com/articles/d41586-025-02097-6
• U.S. National Library of Medicine. PubMed, from https://pubmed.ncbi.nlm.nih.gov, accessed on 2024-09-26.
• Biswas, S. S. (2023). Role of Chat GPT in public health. Annals of Biomedical Engineering, 51(5), 868–869. https://doi.org/10.1007/s10439-023-03172-7
• Miller, G. A. (1995). WordNet: A lexical database for English. Communications of the ACM, 38, 39–41. https://doi.org/10.1145/219717.219748
• Grefenstette, G. (1999). Tokenization. In Voutilainen, A., Heikkilä, J., & Anttila, A. (eds.), Syntactic wordclass tagging, Springer, Dordrecht, pp. 117–133.
• Kaur, J., & Buttar, P. K. (2018). A systematic review on stopword removal algorithms. International Journal on Future Revolution in Computer Science & Communication Engineering, 4(4), 207–210.
• Kannan, S., & Gurusamy, V. (2014). Preprocessing techniques for text mining. International Journal of Computer Science & Communication Networks, 5(1), 7–16.
• Sparck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1), 11–21. https://doi.org/10.1108/eb026526
• Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI), Montreal, Canada, pp. 1137–1145.
• Hosmer Jr, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression. John Wiley & Sons, New York.
• Charbuty, B., & Abdulazeez, A. (2021). Classification based on decision tree algorithm for machine learning. Journal of Applied Science and Technology Trends, 2(1), 20–28. https://doi.org/10.38094/jastt20165
• Breiman, L. (2017). Classification and regression trees. Routledge, New York.
• Kulkarni, V. Y., & Sinha, P. K. (2012). Pruning of random forest classifiers: A survey and future directions. Proceedings of the International Conference on Data Science & Engineering (ICDSE), pp. 64–68. IEEE.
• Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018
• Abdel Hady, M. F., & Schwenker, F. (2010). Combining committee-based semi-supervised learning and active learning. Journal of Computer Science and Technology, 25(4), 681–698. https://doi.org/10.1007/s11390-010-9357-6 dl.acm.org+8
• Han, J., Kamber, M., & Pei, J. (2011). Data Mining: Concepts and Techniques (3rd ed.). Morgan Kaufmann, San Francisco. https://doi.org/10.1016/C2009-0-61819-5
• Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer, New York.
• Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. Cambridge University Press, Cambridge.
• Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30.