Detection of Malicious Codes Generated by Large Language Models: A Comparison of GPT-3.5, GPT-4o, Gemini, and Claude

Year 2025, Volume: 14 Issue: 1, 1 - 12, 25.03.2025

Meltem Kurt Pehlivanoğlu , Murat Görkem Çoban

https://doi.org/10.55859/ijiss.1634763

Abstract

This study presents novel machine learning-based approaches for detecting whether source code generated by Large Language Models (LLMs) contains malicious code. To achieve this, comprehensive datasets comprising malicious and benign code samples were created using the GPT-3.5 (ChatGPT), GPT-4o, Gemini, and Claude language models. The extracted code samples were then processed through CodeBERT, CodeT5, and manual feature extraction techniques before being classified using various machine learning algorithms. Experimental results demonstrate that this approach can effectively detect malicious software in code generated by LLMs. This study makes contributions to software security and represents a crucial step toward preventing the misuse of LLMs for malicious purposes. Moreover, the Random Forest algorithm for binary malicious code classification in LLM-generated code achieved the best F$_{1}$ score of 94.92\% on the ChatGPT-generated dataset (with CodeT5 feature extraction technique). We also showed that the classification models exhibited poor performance on the dataset generated by Claude language model.

Keywords

Large Language Models, Malware Detection, Malicious Code, Machine Learning, CodeBERT, CodeT5, Artificial Intelligence Security

Project Number

The authors are supported by the Scientific and Technological Research Council of Turkey, under grant TUBITAK 2209-A 1919B012324109.

References

• J. Ye, X. Chen, N. Xu, C. Zu, Z. Shao, S. Liu, Y. Cui, Z. Zhou, C. Gong, Y. Shen et al., “A comprehensive capability analysis of gpt-3 and gpt-3.5 series models,” arXiv preprint arXiv:2303.10420, 2023.
• R. Islam and O. M. Moushi, “Gpt-4o: The cutting-edge advancement in multimodal llm,” Authorea Preprints, 2024.
• G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024.
• J. Bae, S. Kwon, and S. Myeong, “Enhancing software code vulnerability detection using gpt-4o and claude-3.5 sonnet: A study on prompt engineering techniques,” Electronics, vol. 13, no. 13, p. 2657, 2024.
• Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pretrained model for programming and natural languages,” arXiv , preprint arXiv:2002.08155, 2020.
• Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifieraware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859, 2021.
• N. S¸ahin, “Malware detection using transformers-based model gpt-2,” Master’s thesis, Middle East Technical University, 2021.
• M. Botacin, “Gpthreats-3: Is automatic malware generation a threat?” in 2023 IEEE Security and Privacy Workshops (SPW). IEEE, 2023, pp. 238–254.
• A. Monje, A. Monje, R. A. Hallman, and G. Cybenko, “Being a bad influence on the kids: Malware generation in less than five minutes using chatgpt,” 2023, unpublished.
• F. Iqbal, F. Samsom, F. Kamoun, and A. MacDermott, “When ´ chatgpt goes rogue: exploring the potential cybersecurity threats of ai-powered conversational chatbots,” Frontiers in Communications and Networks, vol. 4, p. 1220243, 2023.
• N. Begou, J. Vinoy, A. Duda, and M. Korczynski, “Exploring ´ the dark side of ai: Advanced phishing attack design and deployment using chatgpt,” in 2023 IEEE Conference on Communications and Network Security (CNS). IEEE, 2023, pp. 1–6.
• Y. M. Pa Pa, S. Tanizaki, T. Kou, M. Van Eeten, K. Yoshioka, and T. Matsumoto, “An attacker’s dream? exploring the capabilities of chatgpt for developing malware,” in Proceedings of the 16th Cyber Security Experimentation and Test Workshop, 2023, pp. 10–18.
• S. Gonzalez-Carvajal and E. C. Garrido-Merch ´ an, “Comparing ´ bert against traditional machine learning text classification,” arXiv preprint arXiv:2005.13012, 2020.
• K. Mohammadi, “Human vs machine generated text detection in persian,” 2023.
• S. Zevin and C. Holzem, “Machine learning based source code classification using syntax oriented features,” arXiv preprint arXiv:1703.07638, 2017.
• A. Bhandarkar, M. A. DM, D. Vishwachetan, A. Mushtaq, D. Kadam, and S. Saxena, “Unmasking the ai hand: A machine learning approach to deciphering authorship,” in 2024 3rd International Conference for Innovation in Technology (INOCON). IEEE, 2024, pp. 1–6.
• N. Abdoun and M. Chami, “Automatic text classification of pdf documents using nlp techniques,” in INCOSE International Symposium, vol. 32, no. 1. Wiley Online Library, 2022, pp. 1320–1331.