A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection

Abstract

The world has witnessed a fast-paced digital transformation in the past decade, giving rise to all-connected environments. While the increasingly widespread availability of networks has benefited many aspects of our lives, providing the necessary infrastructure for smart autonomous systems, it has also created a large cyber attack surface. This has made real-time network intrusion detection a significant component of any computerized system. With the advances in computer hardware architectures with fast, high-volume data processing capabilities and the developments in the field of artificial intelligence, deep learning has emerged as a significant aid for achieving accurate intrusion detection, especially for zero-day attacks. In this paper, we propose a deep reinforcement learning-based approach for network intrusion detection and demonstrate its efficacy using two publicly available intrusion detection datasets, namely NSL-KDD and UNSW-NB15. The experiment results suggest that deep reinforcement learning has significant potential to provide effective intrusion detection in the increasingly complex networks of the future.

Keywords

• security
• deep reinforcement learning
• intrusion detection

References

1. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra and M. Riedmiller, “Playing Atari with Deep Reinforcement Learning,” pp. 1–9, 2013. Retrieved from http://arxiv.org/abs/1312.5602
2. M. Lai, “Giraffe: Using Deep Reinforcement Learning to Play Chess,” September, 2015. Retrieved from http://arxiv.org/abs/1509.01549
3. L. Wang, D. Zhang, L. Gao, J. Song, L. Guo and H. T. Shen, “MathDQN: Solving arithmetic word problems via deep reinforcement learning,” 32nd AAAI Conference on Artificial Intelligence, pp. 5545–5552, 2018.
4. S. Nemati, M. M. Ghassemi and G. D. Clifford, “Optimal medication dosing from suboptimal clinical examples: A deep reinforcement learning approach,” Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2016.
5. Z. Zhou, X. Li and R. N. Zare, “Optimizing Chemical Reactions with Deep Reinforcement Learning,” ACS Central Science, vol. 3, no. 12, pp. 1337–1344, 2017.
6. M. Mahmud, M. S. Kaiser, A. Hussain and S. Vassanelli, “Applications of Deep Learning and Reinforcement Learning to Biological Data,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 6, pp. 2063–2079, 2018.
7. T. Yen, A. Oprea and K. Onarlioglu, "Beehive: large-scale log analysis for detecting suspicious activity in enterprise networks,". Proc. 29th Annual Computer Security Applications Conference, pp. 199–208, 2013.
8. A. Razaq, H. Tianfield and P. Barrie, "A big data analytics based approach to anomaly detection," Proc. - 2016 IEEE/ACM 3rd International Conference on Big Data Computing Applications and Technologies (BDCAT), pp. 187–193, 2016.

9. A. O. Balogun and R. G. Jimoh, "Anomaly intrusion detection using a hybrid of decision tree and K-nearest neighbor," Journal of Advances in Scientific Research & Applications (JASRA), vol. 2, no. 1, pp. 67-74, 2015.
10. A. Hariharan, A. Gupta and T. Pal, "CAMLPAD: Cybersecurity Autonomous Machine Learning Platform for Anomaly Detection," Proc. Future of Information and Communication Conference (FICC), San Francisco, CA, USA, pp. 705-720, 2020.
11. M. S. Pervez and D. M. Farid, “Feature selection and intrusion classification in NSL-KDD cup 99 dataset employing SVMs,” SKIMA 2014 - 8th International Conference on Software, Knowledge, Information Management and Applications, pp. 1–6, 2014.
12. B. Ingre and A. Yadav, "Performance analysis of NSL-KDD dataset using ANN," 2015 International Conference on Signal Processing and Communication Engineering Systems, pp. 92-96, 2015.
13. S.O.M. Kamel, N. Hegazi, H. Harb, A. ElDein and H. ElKader, "AdaBoost Ensemble Learning Technique for Optimal Feature Subset Selection," International Journal of Computer Networks and Communications Security vol. 4, no. 1, pp. 1–11, 2016.
14. W. Hu, W. Hu, and S. Maybank, "AdaBoost-Based Algorithm for Network Intrusion Detection," IEEE Transactions on Systems, Man, and Cybernetics - Part B: Cybernetics, vol. 38, no. 2, pp. 577-583, 2008.
15. A. H. Engly, A. R. Larsen, and W. Meng, "Evaluation of Anomaly-Based Intrusion Detection with Combined Imbalance Correction and Feature Selection," Proc. 14th International Conference on Network and System Security, Melbourne, Australia, pp. 277-291, 2020.
16. N. Moustafa and J. Slay, "A hybrid feature selection for network intrusion detection systems: central points and association rules," arXiv:1707.05505, (2017) [cs.CR].
17. J. Kim and H. Kim, “Applying Recurrent Neural Network to Intrusion Detection with Hessian Free Optimization,” In: Kim H., Choi D. (eds) Information Security Applications. WISA 2015. Lecture Notes in Computer Science, vol. 9503, 2016, Springer, Cham.
18. Y. Chuan-long, Z. Yue-fei, F. Jin-long and H. Xin-zheng, “A Deep Learning Approach for Intrusion Detection using Recurrent Neural Networks,” IEEE Access, vol. 5, pp. 21954 - 2196, 2017.
19. C. Yin, Y. Zhu, J. Fei, and X. He, ‘‘A deep learning approach for intrusion detection using recurrent neural networks,’’ IEEE Access, vol. 5, pp. 21954–21961, 2017.
20. Z. Li, A. L. G. Rios, G. Xu, and L. Trajkovic, ‘‘Machine learning techniques for classifying network anomalies and intrusions,’’ in Proc. IEEE Int. Symp. Circuits Syst. (ISCAS), pp. 1–5, 2019.
21. S. Behera, A. Pradhan, and R. Dash, “Deep Neural Network Architecture for Anomaly Based Intrusion Detection System,” 5th International Conference on Signal Processing and Integrated Networks (SPIN 2018), pp. 270– 274, 2018.
22. Z. Li, Z. Qin, K. Huang, X. Yang, and S. Ye, ‘‘Intrusion detection using convolutional neural networks for representation learning,’’ in Proc. Int. Conf. Neural Inf. Process. pp. 858–866, 2017.
23. M. Lopez-Martin, B. Carro, A. Sanchez-Esguevillas, and J. Lloret, ‘‘Conditional variational autoencoder for prediction and feature recovery applied to intrusion detection in IoT,’’ Sensors, vol. 17, no. 9, p. 1967, Aug. 2017.
24. F. A. Khan, A. Gumaei, A. Derhab, and A. Hussain, ‘‘TSDL: A twostage deep learning model for efficient network intrusion detection,’’ IEEE Access, vol. 7, pp. 30373–30385, 2019.
25. T. Ma, F. Wang, J. Cheng, Y. Yu, and X. Chen, ‘‘A hybrid spectral clustering and deep neural network ensemble algorithm for intrusion detection in sensor networks,’’ Sensors, vol. 16, no. 10, p. 1701, Oct. 2016.
26. R. Vinayakumar, M. Alazab, K. P. Soman, P. Poornachandran, A. Al-Nemrat, and S. Venkatraman, ‘‘Deep learning approach for intelligent intrusion detection system,’’ IEEE Access, vol. 7, pp. 41525–41550, 2019.
27. N. Gao, L. Gao, Q. Gao, and H. Wang, "An Intrusion Detection Model Based on Deep Belief Networks," Proc. 2nd International Conference on Advanced Cloud and Big Data, Huangshan, China, pp. 247-252, 2014.
28. B. Deokar and A. Hazarnis, “Intrusion Detection System using Log Files and Reinforcement Learning,” International Journal of Computer Applications, vol. 45, no. 1919, pp. 28–35, 2012.
29. A. Servin and D. Kudenko, “Multi-agent reinforcement learning for intrusion detection: A case study and evaluation,” Frontiers in Artificial Intelligence and Applications, vol. 178, pp. 873–874, 2008.
30. R. Elderman, L. J. J. Pater, A. S. Thie, M. M. Drugan and M. A. Wiering, “Adversarial reinforcement learning in a cyber security simulation,” ICAART 2017- Proceedings of the 9th International Conference on Agents and Artificial Intelligence, pp. 559–566, 2017.
31. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, MA, USA: MIT Press, 2016.
32. L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement Learning: A Survey,” Journal of Artificial Intelligence Research, vol. 4, 1996.
33. N. Moustafa, J. Slay, "UNSW-NB15: A Comprehensive Data Set for Network i Intrusion Detection Systems (UNSW-NB15 Network Data Set)," Proceedings of the 2015 IEEE Military Communications and Information Systems Conference (MilCIS), pp. 1–6, 2015.
34. KDD Cup 1999. Avaliable online: https://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html (Accessed on 20 November 2020).
35. 1998 DARPA Intrusion Detection Evaluation Dataset. Available online: https://www.ll.mit.edu/r-d/datasets/1998-darpa-intrusion-detection-evaluation-dataset (Accessed on 20 November 2020).
36. M. Tavallaee, E. Bagheri, W. Lu, and A.A. Ghorbani, “A detailed analysis of the KDD CUP 99 data set,” IEEE Symposium on Computational Intelligence for Security and Defense Applications, pp. 1–6, 2009.
37. H. Koduvely, “Github repository, gym-network_intrusion,” Retrieved from https://github.com/harik68/gym-network_intrusion, 2018.
38. Y. Sun, B. Xue, M. Zhang, and G. G. Yen, “An Experimental Study on Hyper-parameter Optimization for Stacked Auto-Encoders,” Proc. IEEE Congress on Evolutionary Computation, Rio de Janeiro, Brazil, pp. 1-8, 2018.
39. M. Lopez-Martin, B. Carro, A. Sanchez-Esguevillas, and J. Lloret, “Shallow neural network with kernel approximation for prediction problems in highly demanding data networks,” Expert Systems with Applications, vol. 124, pp. 196-208, 2019.
40. Y. Yang, K. Zheng, B. Wu, Y. Yang, and X. Wang, “Network intrusion detection based on supervised adversarial variational auto-encoder with regularization,” IEEE Access, vol. 8., pp. 42169-42184, 2020.