Comparative Study of Transformer-based Architecture and CNN-LSTM Hybrid Model for Network Intrusion Detection Using CSE-CIC-IDS 2018 Dataset

Sarangapani Palani; Praveenkumar S; Prakash Durairaj; Louies Praveen S; Sharun Victor

doi:10.31127/tuje.1843251

Comparative Study of Transformer-based Architecture and CNN-LSTM Hybrid Model for Network Intrusion Detection Using CSE-CIC-IDS 2018 Dataset

Abstract

Network intrusion detection systems (NIDS) are critical for protecting modern network infrastructures from sophisticated cyber threats. This study presents a comprehensive comparative analysis of two advanced deep learning architectures: a pure Transformer-based model and a hybrid Convolutional Neural Network-Long Short-Term Memory (CNN-LSTM) model for multi-class intrusion detection. Using the preprocessed and balanced CSE-CIC-IDS 2018 dataset (Version 1, 2024), we evaluate both models across multiple performance metrics. The dataset underwent rigorous preprocessing including duplicate removal, feature engineering, and two-stage resampling (Random Undersampling and SMOTE) to address class imbalance. Our experimental results demonstrate that the Transformer model achieves 99.42% accuracy with a ROC-AUC of 99.58%, while the CNN-LSTM hybrid model achieves 99.18% accuracy with a ROC-AUC of 99.35%. The Transformer architecture excels in capturing long-range dependencies and parallel processing, while CNN-LSTM effectively combines spatial feature extraction with temporal sequence learning. Both models demonstrate superior performance in detecting minority attack classes compared to traditional approaches. This study provides insights into architectural trade-offs between self-attention mechanisms and recurrent-convolutional hybrids, offering practical guidance for implementing robust intrusion detection systems.

Keywords

References

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30).
Yang, K., Wang, J., & Li, M. (2024). An enhanced IIoT intrusion detection method using BiGRU, attention, and Inception-CNN with hybrid sampling and feature selection. Scientific Reports, 14, 19339. https://doi.org/10.1038/s41598-024-70493-z
Song, Y., Luktarhan, N., Shi, Z., & Wu, H. (2023). TGA: A novel network intrusion detection method based on TCN, BiGRU and attention mechanism. Electronics, 12(13), 2881. https://doi.org/10.3390/electronics12132881
Wang, Y. C., Houng, Y. C., Chen, H. X., & Tseng, S. M. (2023). Network anomaly intrusion detection based on deep learning approach. Sensors, 23(4), 2171. https://doi.org/10.3390/s23042171
Guo, D., & Xie, Y. (2025). Research on network intrusion detection model based on hybrid sampling and deep learning. Sensors, 25(5), 1578. https://doi.org/10.3390/s25051578
Aljabri, J. (2025). Attack-resilient IoT security framework using multi-head attention-based representation learning with improved white shark optimization algorithm. Scientific Reports, 15(1), 14255. https://doi.org/10.1038/s41598-025-86024-4
Imrana, Y., Xiang, Y., Ali, L., Noor, A., Sarpong, K., & Abdullah, M. A. (2024). CNN-GRU-FF: A double-layer feature fusion-based network intrusion detection system using convolutional neural network and gated recurrent units. Complex & Intelligent Systems, 10(3), 3353–3370. https://doi.org/10.1007/s40747-023-01316-6
Han, J., & Pak, W. (2023). Hierarchical LSTM-based network intrusion detection system using hybrid classification. Applied Sciences, 13(5), 3089. https://doi.org/10.3390/app13053089

Udurume, M., Shakhov, V., & Koo, I. (2024). Comparative analysis of deep convolutional neural network—Bidirectional long short-term memory and machine learning methods in intrusion detection systems. Applied Sciences, 14(16), 6967. https://doi.org/10.3390/app14166967
Mohamed, A. (2024). CSE-CIC-IDS2018 [Data set]. Mendeley Data, V1. https://doi.org/10.17632/g8kf837hy6.1
Fernández, A., Garcia, S., Herrera, F., & Chawla, N. V. (2018). SMOTE for learning from imbalanced data: Progress and challenges, marking the 15-year anniversary. Journal of Artificial Intelligence Research, 61, 863–905. https://doi.org/10.1613/jair.1.11192
Gopalan, S. S., Ravikumar, D., Linekar, D., Raza, A., & Hasib, M. (2021). Balancing approaches towards ML for IDS: A survey for the CSE-CIC-IDS dataset. In 2020 International Conference on Communications, Signal Processing, and Their Applications (ICCSPA) (pp. 1–6). IEEE. https://doi.org/10.1109/ICCSPA49915.2021.9385740
Buda, M., Maki, A., & Mazurowski, M. A. (2018). A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106, 249–259. https://doi.org/10.1016/j.neunet.2018.07.011
Abd Elrahman, S. M., & Abraham, A. (2013). A review of class imbalance problem. Journal of Network and Innovative Computing, 1, 332–340.
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. https://doi.org/10.48550/arXiv.1406.1078
Kamal, H., & Mashaly, M. (2024). Advanced hybrid Transformer-CNN deep learning model for effective intrusion detection systems with class imbalance mitigation using resampling techniques. Future Internet, 16(12), 481. https://doi.org/10.3390/fi16120481
Sureshkumar, B., & Senthilkumar, P. (2025). Comparative study of BiGRU with multi-head attention and CNN for network intrusion detection using a cleaned and balanced CSE-CIC-IDS 2018 dataset. Turkish Journal of Engineering, 9(4), 725–737. https://doi.org/10.31127/tuje.1547933
Singh, A. (2025). Real time intrusion detection in edge computing using machine learning techniques. Turkish Journal of Engineering, 9(2), 385–393. https://doi.org/10.31127/tuje.1516046
Sinap, V. (2025). A novel hyperparameter tuning method for enhanced intrusion detection in network security. Turkish Journal of Engineering, 9(3), 519–534. https://doi.org/10.31127/tuje.1624366
Senthilkumar, P., & Kumaresan, D. (2025). Comparative performance analysis of machine learning algorithms: Random cut forest, robust random cut forest, and Amazon SageMaker random cut forest for intrusion detection systems using the CIC IDS 2017 dataset. Turkish Journal of Engineering, 9(3), 535–548. https://doi.org/10.31127/tuje.1614930
Singh, A. P., Singh, M., Bhatia, K., & Pathak, H. (2024). Encrypted malware detection methodology without decryption using deep learning-based approaches. Turkish Journal of Engineering, 8(3), 498–509. https://doi.org/10.31127/tuje.1416933
Hu, Z., Liu, G., Li, Y., & Zhuang, S. (2024). SAGB: Self-attention with gate and BiGRU network for intrusion detection. Complex & Intelligent Systems, 10(6), 8467–8479. https://doi.org/10.1007/s40747-024-01573-z
Cao, B., Li, C., Song, Y., Qin, Y., & Chen, C. (2022). Network intrusion detection model based on CNN and GRU. Applied Sciences, 12(9), 4184. https://doi.org/10.3390/app12094184
Zhang, J., Zhang, X., Liu, Z., Fu, F., Jiao, Y., & Xu, F. (2023). A network intrusion detection model based on BiLSTM with multi-head attention mechanism. Electronics, 12(19), 3984. https://doi.org/10.3390/electronics12193984
Benchama, A., Zebbara, K., Elasri, S., & Aftatah, M. (2024). Optimized CNN-BiGRU intrusion detection model with SMOTE enhancement: Using Optuna for automated hyperparameter tuning. In The International Workshop on Big Data and Business Intelligence (pp. 66–76). Springer Nature. https://doi.org/10.1007/978-3-031-68675-7_6
Xin, Y., Kong, L., Liu, Z., Chen, Y., Li, Y., Zhu, H., & Wang, C. (2018). Machine learning and deep learning methods for cybersecurity. IEEE Access, 6, 35365–35381. https://doi.org/10.1109/ACCESS.2018.2836950
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. https://doi.org/10.48550/arXiv.1810.04805
Senthilkumar, P., & Suresh Kumar, B. (2025). Advancing multi-class intrusion detection: A comparative evaluation of LSTM and Bi-LSTM on class-imbalanced CIC-IDS-2017. Turkish Journal of Engineering, 9(3), 578–590. https://doi.org/10.31127/tuje.1658662

Details

Primary Language

English

Subjects

Computer System Software

Journal Section

Research Article

Authors

Sarangapani Palani ^*
0000-0002-8684-7557
India

Praveenkumar S
0000-0002-8594-1429
India

Prakash Durairaj
0009-0002-8320-0763
India

Louies Praveen S
0000-0002-9468-3038
India

Sharun Victor
0000-0002-5371-2410
India

Publication Date

July 6, 2026

Submission Date

December 30, 2025

Acceptance Date

June 2, 2026

Published in Issue

Year 2026 Volume: 10 Number: 3

DOI

https://doi.org/10.31127/tuje.1843251

IZ

https://izlik.org/JA54MG65WU

Cite

RIS / Bibtex

APA

Palani, S., S, P., Durairaj, P., Praveen S, L., & Victor, S. (2026). Comparative Study of Transformer-based Architecture and CNN-LSTM Hybrid Model for Network Intrusion Detection Using CSE-CIC-IDS 2018 Dataset. Turkish Journal of Engineering, 10(3), 1203-1219. https://doi.org/10.31127/tuje.1843251

AMA

1.Palani S, S P, Durairaj P, Praveen S L, Victor S. Comparative Study of Transformer-based Architecture and CNN-LSTM Hybrid Model for Network Intrusion Detection Using CSE-CIC-IDS 2018 Dataset. TUJE. 2026;10(3):1203-1219. doi:10.31127/tuje.1843251

Chicago

Palani, Sarangapani, Praveenkumar S, Prakash Durairaj, Louies Praveen S, and Sharun Victor. 2026. “Comparative Study of Transformer-Based Architecture and CNN-LSTM Hybrid Model for Network Intrusion Detection Using CSE-CIC-IDS 2018 Dataset”. Turkish Journal of Engineering 10 (3): 1203-19. https://doi.org/10.31127/tuje.1843251.

EndNote

Palani S, S P, Durairaj P, Praveen S L, Victor S (July 1, 2026) Comparative Study of Transformer-based Architecture and CNN-LSTM Hybrid Model for Network Intrusion Detection Using CSE-CIC-IDS 2018 Dataset. Turkish Journal of Engineering 10 3 1203–1219.

IEEE

[1]S. Palani, P. S, P. Durairaj, L. Praveen S, and S. Victor, “Comparative Study of Transformer-based Architecture and CNN-LSTM Hybrid Model for Network Intrusion Detection Using CSE-CIC-IDS 2018 Dataset”, TUJE, vol. 10, no. 3, pp. 1203–1219, July 2026, doi: 10.31127/tuje.1843251.

ISNAD

Palani, Sarangapani - S, Praveenkumar - Durairaj, Prakash - Praveen S, Louies - Victor, Sharun. “Comparative Study of Transformer-Based Architecture and CNN-LSTM Hybrid Model for Network Intrusion Detection Using CSE-CIC-IDS 2018 Dataset”. Turkish Journal of Engineering 10/3 (July 1, 2026): 1203-1219. https://doi.org/10.31127/tuje.1843251.

JAMA

1.Palani S, S P, Durairaj P, Praveen S L, Victor S. Comparative Study of Transformer-based Architecture and CNN-LSTM Hybrid Model for Network Intrusion Detection Using CSE-CIC-IDS 2018 Dataset. TUJE. 2026;10:1203–1219.

MLA

Palani, Sarangapani, et al. “Comparative Study of Transformer-Based Architecture and CNN-LSTM Hybrid Model for Network Intrusion Detection Using CSE-CIC-IDS 2018 Dataset”. Turkish Journal of Engineering, vol. 10, no. 3, July 2026, pp. 1203-19, doi:10.31127/tuje.1843251.

Vancouver

1.Sarangapani Palani, Praveenkumar S, Prakash Durairaj, Louies Praveen S, Sharun Victor. Comparative Study of Transformer-based Architecture and CNN-LSTM Hybrid Model for Network Intrusion Detection Using CSE-CIC-IDS 2018 Dataset. TUJE. 2026 Jul. 1;10(3):1203-19. doi:10.31127/tuje.1843251