A novel machine learning-based artificial intelligence approach for log analysis using blockchain technology

Abstract

Cybercrime is one of the fastest-growing crimes worldwide. It is observed that every seven seconds, cyber attackers penetrate cyber systems. While detecting an anomaly or attack, the log system is one of the crucial components of any system storing and managing all the events. It has always been challenging to detect an anomaly in logs. This is because of continuous and ever-changing log events and their mutability property. In this paper, we develop a ma-chine learning-based artificial intelligence approach to address this issue of log analysis by proposing two modules. The first one is anomaly detection using different machine learning models. The second one is a distributed immutable storage system for securely storing the logs. In addition, we present a descriptive and user-friendly web application by integrating all modules using HTML, CSS, and Flask Framework on the Heroku cloud environment. The re-sults demonstrate that the proposed hybrid machine learning models are capable of achieving 99.7% accuracy in detecting network anomalies.

Keywords

• Immutable Storage
• Isolation Forest Machine Learning
• Logs
• KNN

References

1. REFERENCES
2. [1] Simoes V, Maniar H, Abubakar A, Zhao T. Deep learning for multiwell automatic log correction. In: SPWLA 63rd Annual Logging Symposium. OnePetro; 2022. [CrossRef]
3. [2] Oliner A, Ganapathi A, Xu W. Advances and challenges in log analysis. Commun ACM 2012;55:5561. [CrossRef]
4. [3] Albahar M, Alansari D, Jurcut A. An empirical comparison of pen-testing tools for detecting web app vulnerabilities. Electronics 2022;11:2991. [CrossRef]
5. [4] Candel JMO, Gimeno FJM, Mora Mora H. Serverless security analysis for IoT applications. In: International Conference on Ubiquitous Computing and Ambient Intelligence. Springer; 2023. p. 393400. [CrossRef]
6. [5] Zhu J, He S, Liu J, He P, Xie Q, Zheng Z, et al. Tools and benchmarks for automated log parsing. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE; 2019. p. 121130. [CrossRef]
7. [6] Behera A, Panigrahi CR, Pati B. Unstructured log analysis for system anomaly detection—A study. In: Advances in Data Science and Management. Springer; 2022. p. 497509. [CrossRef]
8. [7] Chen QX, Chang XH. Resilient filter of nonlinear network systems with dynamic event-triggered mechanism and hybrid cyber attack. Appl Math Comput 2022;434:127419. [CrossRef]

9. [8] Rout B, Natarajan B. Impact of cyber-attacks on distributed compressive sensing based state estimation in power distribution grids. Int J Electr Power Energy Syst 2022;142:108295. [CrossRef]
10. [9] Ghiasi M, Niknam T, Wang Z, Mehrandezh M, Dehghani M, Ghadimi N. A comprehensive review of cyber-attacks and defense mechanisms for improving security in smart grid energy systems: Past, present and future. Electr Power Syst Res 2023;215:108975. [CrossRef]
11. [10] Abdullahi M, Baashar Y, Alhussian H, Alwadain A, Aziz N, Capretz LF, et al. Detecting cybersecurity attacks in Internet of Things using artificial intelligence methods: A systematic literature review. Electronics 2022;11:198. [CrossRef]
12. [11] Zhang X, Xu Y, Lin Q, Qiao B, Zhang H, Dang Y, et al. Robust log-based anomaly detection on unstable log data. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering; 2019. p. 807817. [CrossRef]
13. [12] Wang B, Ying S, Yang Z. A log-based anomaly detection method with efficient neighbor searching and automatic K neighbor selection. Sci Program 2020;2020:117. [CrossRef]
14. [13] Wang J, Tang Y, He S, Zhao C, Sharma PK, Alfarraj O, et al. LogEvent2vec: Log event-to-vector based anomaly detection for large-scale logs in Internet of Things. Sensors 2020;20:2451. [CrossRef]
15. [14] Wang Z, Tian J, Fang H, Chen L, Qin J. LightLog: A lightweight temporal convolutional network for log anomaly detection on the edge. Comput Netw 2022;203:108616. [CrossRef]
16. [15] Catillo M, Pecchia A, Villano U. AutoLog: Anomaly detection by deep auto encoding of system logs. Expert Syst Appl 2022;191:116263. [CrossRef]
17. [16] Pourmajidi W, Miranskyy A. Logchain: Blockchain-assisted log storage. In: 2018 IEEE 11th International Conference on Cloud Computing (CLOUD). IEEE; 2018. p. 978982. [CrossRef]
18. [17] Pourmajidi W, Zhang L, Steinbacher J, Erwin T, Miranskyy A. Immutable log storage as a service. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE; 2019. p. 280281. [CrossRef]
19. [18] Oprea SV, Bâra A. Machine learning classification algorithms and anomaly detection in conventional meters and Tunisian electricity consumption large datasets. Comput Electr Eng 2021;94:107329. [CrossRef]
20. [19] Huang W. A blockchain-based framework for secure log storage. In: 2019 IEEE 2nd International Conference on Computer and Communication Engineering Technology (CCET). IEEE; 2019. p. 96100. [CrossRef]
21. [20] Wang H, Yang D, Duan N, Guo Y, Zhang L. Medusa: Blockchain powered log storage system. In: 2018 IEEE 9th International Conference on Software Engineering and Service Science (ICSESS). IEEE; 2018. p. 518521. [CrossRef]
22. [21] Friedberg I, Skopik F, Settanni G, Fiedler R. Combating advanced persistent threats: From network event correlation to incident detection. Comput Secur 2015;48:3557. [CrossRef]
23. [22] Ambre A, Shekokar N. Insider threat detection using log analysis and event correlation. Procedia Comput Sci 2015;45:436445. [CrossRef]
24. [23] Reguieg H, Benatallah B, Nezhad HRM, Toumani F. Event correlation analytics: Scaling process mining using mapreduce-aware event correlation discovery techniques. IEEE Trans Serv Comput 2015;8:847860. [CrossRef]
25. [24] Bračevac O, Amin N, Salvaneschi G, Erdweg S, Eugster P, Mezini M. Versatile event correlation with algebraic effects. Proc ACM Program Lang. 2018;2(ICFP):131. [CrossRef]
26. [25] Kotenko IV, Levshun DS, Chechulin AA. Event correlation in the integrated cyber-physical security system. In: 2016 XIX IEEE International Conference on Soft Computing and Measurements (SCM). IEEE; 2016. p. 484486. [CrossRef]
27. [26] Landauer M, Skopik F, Wurzenberger M, Hotwagner W, Rauber A. Have it your way: Generating customized log datasets with a model-driven simulation testbed. IEEE Trans Reliab 2020;70:402415. [CrossRef]
28. [27] Rahman RU, Tomar DS. New biostatistics features for detecting web bot activity on web applications. Comput Secur 2020;97:102001. [CrossRef]
29. [28] Breier J, Branišová J. A dynamic rule creation based anomaly detection method for identifying security breaches in log records. Wirel Pers Commun 2017;94:497511. [CrossRef]
30. [29] Rahman RU, Tomar DS. Threats of price scraping on e-commerce websites: Attack model and its detection using neural network. J Comput Virol Hack Tech 2021;17:7589. [CrossRef]
31. [30] Alkawaz MH, Steven SJ, Hajamydeen AI, Ramli R. A comprehensive survey on identification and analysis of phishing website based on machine learning methods. In: 2021 IEEE 11th IEEE Symposium on Computer Applications & Industrial Electronics (ISCAIE). IEEE; 2021. p. 8287. [CrossRef]