Balkan Journal of Electrical and Computer Engineering 2147-284X 2147-284X MUSA YILMAZ 10.17694/bajece.1506554 Software Testing, Verification and Validation Yazılım Testi, Doğrulama ve Validasyon Analysis of Malicious Files Gathering via Honeypot Trap System and Benchmark of Anti-Virus Software https://orcid.org/0000-0003-0175-5045 Başer Melike İSTANBUL ÜNİVERSİTESİ-CERRAHPAŞA https://orcid.org/0000-0002-7587-3127 Güven Ebu Yusuf İSTANBUL ÜNİVERSİTESİ-CERRAHPAŞA https://orcid.org/0000-0002-1846-6090 Aydın Muhammed Ali İSTANBUL ÜNİVERSİTESİ-CERRAHPAŞA 01 07 2025 12 4 337 348 07 01 2024 11 22 2024 Copyright © 2013, Balkan Journal of Electrical and Computer Engineering 2013 Balkan Journal of Electrical and Computer Engineering

In the age of widespread digital integration, the rise in cyber threats is evident. Cyber attackers use malicious software (malware) to compromise data and exploit system resources, employing tactics such as remote control or ransom through data encryption. Despite the common use of antivirus software with signature-based detection, this study reveals its limitations. Using a honeypot trap system on Google Cloud, suspicious files uploaded by attackers were analyzed. Results from evaluating these files with 64 antivirus programs show that relying solely on signature-based methods is insufficient. Only three programs had success rates exceeding 90\%, while the majority had success rates predominantly below 70\%. This underscores the need for diverse detection techniques alongside signature-based methods to enhance cybersecurity. The repository containing collected malicious files and the Python script is available on Github, serving as a valuable research resource for further exploration.

 Malware Honeypot Antivirus Benchmark Signature-based Malware Dataset [1] G. Pitolli, G. Laurenza, L. Aniello, L. Querzoni, and R. Baldoni, “Malfamaware: automatic family identification and malware classification through online clustering,” International Journal of information security, vol. 20, pp. 371–386, 2021. [2] M. Amal and P. Venkadesh, “Review of cyber attack detection: Honeypot system,” Webology, vol. 19, no. 1, pp. 5497–5514, 2022. [3] S. COOK, “Malware statistics in 2022: Frequency, impact, cost & more,” Feb 2022. [Online]. Available: https: //www.comparitech.com/antivirus/malware-statistics-facts/ [4] S. S. Chakkaravarthy, D. Sangeetha, and V. Vaidehi, “A survey on malware analysis and mitigation techniques,” Computer Science Review, vol. 32, pp. 1–23, 2019. [5] N. Pachhala, S. Jothilakshmi, and B. P. Battula, “A comprehensive survey on identification of malware types and malware classification using machine learning techniques,” in 2021 2nd International Conference on Smart Electronics and Communication (ICOSEC). IEEE, 2021, pp. 1207–1214. [6] C. Rohith and G. Kaur, “A comprehensive study on malware detection and prevention techniques used by anti-virus,” in 2021 2nd International Conference on Intelligent Engineering and Management (ICIEM). IEEE, 2021, pp. 429–434. [7] K. Oosthoek and C. Doerr, “Cyber threat intelligence: A product without a process?” International Journal of Intelligence and CounterIntelligence, vol. 34, no. 2, pp. 300–315, 2021. [8] D. Aygor and E. Aktan, “The limitations of signature-based ¨ and dynamic analysis methods in detecting malwares: A case study,” Journal of the Faculty of Engineering and Architecture of Gazi University, vol. 37, no. 1, pp. 305–315, 2022. [9] U. Inayat, M. F. Zia, F. Ali, S. M. Ali, H. M. A. Khan, and W. Noor, “Comprehensive review of malware detection techniques,” in 2021 International Conference on Innovative Computing (ICIC). IEEE, 2021, pp. 1–6. [10] D. Laka, “Malware: Types, analysis and classification,” Analysis and Classification (January 14, 2022), 2022. [11] E. Tekiner, A. Acar, A. S. Uluagac, E. Kirda, and A. A. Selcuk, “Sok: cryptojacking malware,” in 2021 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2021, pp. 120–139. [12] S. Talukder and Z. Talukder, “A survey on malware detection and analysis tools,” International Journal of Network Security & Its Applications (IJNSA) Vol, vol. 12, 2020. [13] S. A. Roseline and S. Geetha, “A comprehensive survey of tools and techniques mitigating computer and mobile malware attacks,” Computers & Electrical Engineering, vol. 92, p. 107143, 2021. [14] S. Varlioglu, N. Elsayed, Z. ElSayed, and M. Ozer, “The dangerous combo: Fileless malware and cryptojacking,” SoutheastCon 2022, pp. 125–132, 2022. [15] T. Alsmadi and N. Alqudah, “A survey on malware detection techniques,” in 2021 International Conference on Information Technology (ICIT). IEEE, 2021, pp. 371–376. [16] A. Chavan, K. Kerakalamatti, and S. Srivastva, “Implementation of portable antivirus system using signature-based detection and heuristic analysis,” in 2021 5th International Conference on Trends in Electronics and Informatics (ICOEI). IEEE, 2021, pp. 1481–1486. [17] M. Botacin, M. Z. Alves, D. Oliveira, and A. Gregio, “Heaven: ´ A hardware-enhanced antivirus engine to accelerate real-time, signature-based malware detection,” Expert Systems with Applications, vol. 201, p. 117083, 2022. [18] M. J. H. Faruk, H. Shahriar, M. Valero, F. L. Barsha, S. Sobhan, M. A. Khan, M. Whitman, A. Cuzzocrea, D. Lo, A. Rahman et al., “Malware detection and prevention using artificial intelligence techniques,” in 2021 IEEE International Conference on Big Data (Big Data). IEEE, 2021, pp. 5369– 5377. [19] S. M. de Lima, H. K. d. L. Silva, J. H. d. S. Luz, H. J. d. N. Lima, S. L. d. P. Silva, A. de Andrade, and A. M. da Silva, “Artificial intelligence-based antivirus in order to detect malware preventively,” Progress in Artificial Intelligence, vol. 10, no. 1, pp. 1–22, 2021. [20] S. Rani, K. Tripathi, Y. Arora, and A. Kumar, “Analysis of anomaly detection of malware using knn,” in 2022 2nd International Conference on Innovative Practices in Technology and Management (ICIPTM), vol. 2. IEEE, 2022, pp. 774–779. [21] A. Katkar, S. Shukla, D. Shaikh, and P. Dange, “Malware intrusion detection for system security,” in 2021 International Conference on Communication information and Computing Technology (ICCICT). IEEE, 2021, pp. 1–5. [22] T. A. Assegie, “An optimized knn model for signature-based malware detection,” Tsehay Admassu Assegie.” An Optimized KNN Model for Signature-Based Malware Detection”. International Journal of Computer Engineering In Research Trends (IJCERT), ISSN, pp. 2349–7084, 2021. [23] M. Zyout, R. Shatnawi, and H. Najadat, “Malware classification approaches utilizing binary and text encoding of permissions,” International Journal of Information Security, pp. 1–26, 2023. [24] V. Sethia and A. Jeyasekar, “Malware capturing and analysis using dionaea honeypot,” in 2019 International Carnahan Conference on Security Technology (ICCST). IEEE, 2019, pp. 1–4. [25] I. M. M. Matin and B. Rahardjo, “A framework for collecting and analysis pe malware using modern honey network (mhn),” in 2020 8th International Conference on Cyber and IT Service Management (CITSM). IEEE, 2020, pp. 1–5. [26] A. Kyriakou and N. Sklavos, “Container-based honeypot deployment for the analysis of malicious activity,” in 2018 Global Information Infrastructure and Networking Symposium (GIIS). IEEE, 2018, pp. 1–4. [27] C. Moore, “Detecting ransomware with honeypot techniques,” in 2016 Cybersecurity and Cyberforensics Conference (CCC). IEEE, 2016, pp. 77–81. [28] B. Wang, Y. Dou, Y. Sang, Y. Zhang, and J. Huang, “Iotcmal: Towards a hybrid iot honeypot for capturing and analyzing malware,” in ICC 2020-2020 IEEE International Conference on Communications (ICC). IEEE, 2020, pp. 1–7. [29] J. Aycock, Computer viruses and malware. Springer Science & Business Media, 2006, vol. 22. [30] R. Ball, “Computer viruses, computer worms, and the selfreplication of programs,” in Viruses in all Dimensions: How an Information Code Controls Viruses, Software and Microorganisms. Springer, 2023, pp. 73–85. [31] M. N. Alenezi, H. Alabdulrazzaq, A. A. Alshaher, and M. M. Alkharang, “Evolution of malware threats and techniques: a review,” International Journal of Communication Networks and Information Security, vol. 12, no. 3, pp. 326–337, 2020. [32] “CNSSI 4009: Committee on national security systems (cnss) glossary,” Committee on National Security Systems (CNSS), 2015, accessed: 2024-10-28. [Online]. Available: https://rmf.org/wp-content/uploads/2017/10/CNSSI-4009.pdf [33] J. Aycock, Spyware and adware. Springer Science & Business Media, 2010, vol. 50. [34] I. Kuzminykh and M. Yevdokymenko, “Analysis of security of rootkit detection methods,” in 2019 IEEE International Conference on Advanced Trends in Information Theory (ATIT). IEEE, 2019, pp. 196–199. [35] N. A. Mims, “Chapter 14 - the botnet problem,” in Computer and Information Security Handbook (Fourth Edition), J. R. Vacca, Ed. Morgan Kaufmann, 2025, pp. 261–272. [36] M. Swanson and B. Guttman, “NIST SP 800-12 Rev. 1: An Introduction to Information Security,” National Institute of Standards and Technology (NIST), Tech. Rep. 800-12 Rev. 1, 2017, accessed: 2024-10-28. [Online]. Available: https://csrc.nist.gov/pubs/sp/800/12/r1/final [37] A. Warikoo, “Perspective chapter: Ransomware,” in MalwareDetection and Defense. IntechOpen, 2023. [38] E. Salimi and N. Arastouie, “Backdoor detection system using artificial neural network and genetic algorithm,” in 2011 International Conference on Computational and Information Sciences, 2011, pp. 817–820. [39] H. W. Kim, “A study on countermeasures by detecting trojantype downloader/dropper malicious code,” International Journal of Advanced Culture Technology, vol. 9, no. 4, pp. 288– 294, 2021. [40] A. Damodaran, F. D. Troia, C. A. Visaggio, T. H. Austin, and M. Stamp, “A comparison of static, dynamic, and hybrid analysis for malware detection,” Journal of Computer Virology and Hacking Techniques, vol. 13, no. 1, pp. 1–12, 2017. [41] O. A. Aslan and R. Samet, “A comprehensive review on ¨ malware detection approaches,” IEEE Access, vol. 8, pp. 6249– 6271, 2020. [42] Z. Bazrafshan, H. Hashemi, S. M. H. Fard, and A. Hamzeh, “A survey on heuristic malware detection techniques,” in The 5th Conference on Information and Knowledge Technology, 2013, pp. 113–120. [43] Y. K. B. M. Yunus and S. B. Ngah, “Review of hybrid analysis technique for malware detection,” IOP Conference Series: Materials Science and Engineering, vol. 769, no. 1, p. 012075, feb 2020. [Online]. Available: https://doi.org/10.1088/1757-899x/769/1/012075 [44] R. Sihwail, K. Omar, and K. A. Z. Ariffin, “An effective memory analysis for malware detection and classification,” Comput., Mater. Continua, vol. 67, no. 2, pp. 2301–2320, 2021. [45] K. Monnappa, Learning Malware Analysis: Explore the concepts, tools, and techniques to analyze and investigate Windows malware. Packt Publishing Ltd, 2018. [46] O. Or-Meir, N. Nissim, Y. Elovici, and L. Rokach, “Dynamic malware analysis in the modern era—a state of the art survey,” ACM Computing Surveys (CSUR), vol. 52, no. 5, pp. 1–48, 2019. [47] Y. K. B. M. Yunus and S. B. Ngah, “Review of hybrid analysis technique for malware detection,” in IOP Conference Series: Materials Science and Engineering. IOP Publishing, 2020, p. 012075. [48] R. Sihwail, K. Omar, and K. Z. Ariffin, “A survey on malware analysis techniques: Static, dynamic, hybrid and memory analysis,” Int. J. Adv. Sci. Eng. Inf. Technol, vol. 8, no. 4-2, pp. 1662–1671, 2018. [49] M. Bas¸er, E. Y. Guven, and M. A. Aydın, “Ssh and telnet pro- ¨ tocols attack analysis using honeypot technique:* analysis of ssh and telnet honeypot,” in 2021 6th International Conference on Computer Science and Engineering (UBMK). IEEE, 2021, pp. 806–811. [50] R. Masri and M. Aldwairi, “Automated malicious advertisement detection using virustotal, urlvoid, and trendmicro,” in 2017 8th International Conference on Information and Communication Systems (ICICS). IEEE, 2017, pp. 336–341. [51] A. Salem, S. Banescu, and A. Pretschner, “Maat: Automatically analyzing virustotal for accurate labeling and effective malware detection,” ACM Transactions on Privacy and Security (TOPS), vol. 24, no. 4, pp. 1–35, 2021.