Detection and Analysis of Malicious Software Using Machine Learning Models

Abstract

The continuous evolution of malware poses a significant challenge in cybersecurity, adapting to technological advancements despite implemented security measures. This paper introduces an innovative approach to enhance the detection of obfuscated malware through the integration of machine learning (ML). Utilizing a real-world dataset of prevalent malware types such as spyware, ransomware, and trojan horses, our study addresses the evolving challenges of cybersecurity. In this study, we evaluate the performance of ML algorithms for obfuscated malware detection using the CIC-MalMem-2022 dataset. Our analysis encompasses binary and multi-class classification tasks under various experimental conditions, including percentage splits and 10-fold cross-validation. The evaluated algorithms include Random Tree (RT), Random Forest (RF), J-48 (C4.5), Naive Bayes (NB), and XGBoost. Experimental results demonstrate the effectiveness of RF, J-48, and XGBoost in achieving high accuracy rates across different classification tasks. NB also shows competitive performance but faces challenges in handling imbalanced datasets and multi-class classification. Our findings highlight the importance of employing advanced ML techniques for enhancing obfuscated malware detection capabilities and provide valuable insights for cybersecurity practitioners and researchers. Future research directions include fine-tuning model hyperparameters, exploring ensemble learning approaches, and expanding evaluation to diverse datasets and real-world scenarios.

Keywords

• Information security
• Software analysis
• Malware detection system
• Machine learning

References

1. [1] T. Carrier, P. Victor, A. Tekeoglu, and A. Habibi Lashkari, “Detecting Obfuscated Malware using Memory Feature Engineering,” in International Conference on Information Systems Security and Privacy, 2022. doi: 10.5220/0010908200003120.
2. [2] Z. A. El Houda, “Cyber Threat Actors Review: Examining the Tactics and Motivations of Adversaries in the Cyber Landscape,” in Cyber Security for Next-Generation Computing Technologies, 2024. doi: 10.1201/9781003404361-5.
3. [3] Y. Li, Z. Liu, X. Guan, Z. Wang, X. Guo, and S. Wang, “Hierarchical Obfuscation Malware Detection Method Based on Deep Learning,” in EEI 2022 - 4th International Conference on Electronic Engineering and Informatics, 2022.
4. [4] M. R. Ghazi and N. S. Raghava, “Machine Learning Based Obfuscated Malware Detection in the Cloud Environment with Nature-Inspired Feature Selection,” in 2022 5th International Conference on Multimedia, Signal Processing and Communication Technologies, IMPACT 2022, 2022. doi: 10.1109/IMPACT55510.2022.10029271.
5. [5] M. A. Hossain and M. S. Islam, “Enhanced detection of obfuscated malware in memory dumps: a machine learning approach for advanced cybersecurity,” Cybersecurity, vol. 7, no. 1, 2024, doi: 10.1186/s42400-024-00205-z.
6. [6] B. Janet, A. Nikam, and J. A. Kumar R, “Real Time Malicious URL Detection on twitch using Machine Learning,” in Proceedings of the International Conference on Electronics and Renewable Systems, ICEARS 2022, 2022. doi: 10.1109/ICEARS53579.2022.9751862.
7. [7] M. Hakimi, E. Ahmady, A. K. Shahidzay, A. W. Fazil, M. M. Quchi, and R. Akbari, “Securing Cyberspace: Exploring the Efficacy of SVM (Poly, Sigmoid) and ANN in Malware Analysis,” Cognizance Journal of Multidisciplinary Studies, vol. 3, no. 12, 2023, doi: 10.47760/cognizance.2023.v03i12.017.
8. [8] S. Altaha and K. Riad, “Machine Learning in Malware Analysis: Current Trends and Future Directions,” International Journal of Advanced Computer Science and Applications, vol. 15, no. 1, 2024, doi: 10.14569/IJACSA.2024.01501124.

9. [9] V. Vijayaraj, M. Balamurugan, and M. Oberai, “Machine learning approaches to identify the data types in big data environment: An overview,” The Scientific Temper, vol. 14, no. 03, 2023, doi: 10.58414/scientifictemper.2023.14.3.60.
10. [10] M. Azeem, D. Khan, S. Iftikhar, S. Bawazeer, and M. Alzahrani, “Analyzing and comparing the effectiveness of malware detection: A study of machine learning approaches,” Heliyon, vol. 10, no. 1, 2024, doi: 10.1016/j.heliyon.2023.e23574.
11. [11] A. Nugraha and J. Zeniarja, “Malware Detection Using Decision Tree Algorithm Based on Memory Features Engineering,” Journal of Applied Intelligent System, vol. 7, no. 3, 2022, doi: 10.33633/jais.v7i3.6735.
12. [12] Akoh Atadoga, Enoch Oluwademilade Sodiya, Uchenna Joseph Umoga, and Olukunle Oladipupo Amoo, “A comprehensive review of machine learning’s role in enhancing network security and threat detection,” World Journal of Advanced Research and Reviews, vol. 21, no. 2, 2024, doi: 10.30574/wjarr.2024.21.2.0501.
13. [13] L. Zhang and Q. Yan, “Detect malicious websites by building a neural network to capture global and local features of websites,” Comput Secur, vol. 137, 2024, doi: 10.1016/j.cose.2023.103641.
14. [14] M. Kozák, M. Jureček, M. Stamp, and F. Di Troia, “Creating valid adversarial examples of malware,” Journal of Computer Virology and Hacking Techniques, 2024, doi: 10.1007/s11416-024-00516-2.
15. [15] K. Shaukat, S. Luo, and V. Varadharajan, “A novel machine learning approach for detecting first-time-appeared malware,” Eng Appl Artif Intell, vol. 131, 2024, doi: 10.1016/j.engappai.2023.107801.
16. [16] M. Dener, G. Ok, and A. Orman, “Malware Detection Using Memory Analysis Data in Big Data Environment,” Applied Sciences (Switzerland), vol. 12, no. 17, 2022, doi: 10.3390/app12178604.
17. [17] M. A. Hossain et al., “AI-enabled approach for enhancing obfuscated malware detection: a hybrid ensemble learning with combined feature selection techniques,” International Journal of System Assurance Engineering and Management, 2024, doi: 10.1007/s13198-024-02294-y.
18. [18] D. Cevallos-Salas, F. Grijalva, J. Estrada-Jimenez, D. Benitez, and R. Andrade, “Obfuscated Privacy Malware Classifiers Based on Memory Dumping Analysis,” IEEE Access, vol. 12, 2024, doi: 10.1109/ACCESS.2024.3358840.
19. [19] S. S. Shafin, G. Karmakar, and I. Mareels, “Obfuscated Memory Malware Detection in Resource-Constrained IoT Devices for Smart City Applications,” Sensors, vol. 23, no. 11, 2023, doi: 10.3390/s23115348.
20. [20] L. N. Tidjon and F. Khomh, “Reliable malware analysis and detection using topology data analysis,” arXiv preprint arXiv:2211.01535, 2022.
21. [21] H. Naeem, S. Dong, O. J. Falana, and F. Ullah, “Development of a deep stacked ensemble with process based volatile memory forensics for platform independent malware detection and classification,” Expert Syst Appl, vol. 223, 2023, doi: 10.1016/j.eswa.2023.119952.
22. [22] M. Al-Qudah, Z. Ashi, M. Alnabhan, and Q. Abu Al-Haija, “Effective One-Class Classifier Model for Memory Dump Malware Detection,” Journal of Sensor and Actuator Networks, vol. 12, no. 1, 2023, doi: 10.3390/jsan12010005.
23. [23] D. Smith, S. Khorsandroo, and K. Roy, “Supervised and Unsupervised Learning Techniques Utilizing Malware Datasets,” in 2023 IEEE 2nd International Conference on AI in Cybersecurity, ICAIC 2023, 2023. doi: 10.1109/ICAIC57335.2023.10044169.
24. [24] A. Mezina and R. Burget, “Obfuscated malware detection using dilated convolutional network,” in International Congress on Ultra Modern Telecommunications and Control Systems and Workshops, 2022. doi: 10.1109/ICUMT57764.2022.9943443.
25. [25] K. S. Roy, T. Ahmed, P. B. Udas, M. E. Karim, and S. Majumdar, “MalHyStack: A hybrid stacked ensemble learning framework with feature engineering schemes for obfuscated malware analysis,” Intelligent Systems with Applications, vol. 20, 2023, doi: 10.1016/j.iswa.2023.200283.
26. [26] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic minority over-sampling technique,” Journal of Artificial Intelligence Research, vol. 16, 2002, doi: 10.1613/jair.953.
27. [27] A. H. Lashkari, B. Li, T. L. Carrier, and G. Kaur, “VolMemLyzer: Volatile Memory Analyzer for Malware Classification using Feature Engineering,” in 2021 Reconciling Data Analytics, Automation, Privacy, and Security: A Big Data Challenge, RDAAPS 2021, 2021. doi: 10.1109/RDAAPS48126.2021.9452028.
28. [28] E. Frank, M. A. Hall, and I. H. Witten, “The WEKA Workbench Data Mining: Practical Machine Learning Tools and Techniques,” Morgan Kaufmann, Fourth Edition, 2016.
29. [29] K. Roshan and A. Zafar, “Ensemble adaptive online machine learning in data stream: a case study in cyber intrusion detection system,” International Journal of Information Technology (Singapore), 2024, doi: 10.1007/s41870-024-01727-y.
30. [30] M. M. Abualhaj, A. A. Abu-Shareha, Q. Y. Shambour, A. Alsaaidah, S. N. Al-Khatib, and M. Anbar, “Customized K-nearest neighbors’ algorithm for malware detection,” International Journal of Data and Network Science, vol. 8, no. 1, 2024, doi: 10.5267/j.ijdns.2023.9.012.