Learning from the Normal: Anomaly-Based Intrusion Detection Using Isolation Forest, LOF, and One-Class SVM

Öz

This study presents a comparative analysis of three widely adopted unsupervised anomaly detection algorithms—Isolation Forest, Local Outlier Factor (LOF), and One-Class Support Vector Machine (SVM)—with the aim of evaluating their effectiveness in detecting network intrusions. Using a publicly available cybersecurity dataset, this study applied Principal Component Analysis (PCA) to reduce dimensionality and optimize computational performance. Each model was trained exclusively on normal traffic data and was tested against mixed data containing both normal and attack instances. The performance was assessed using key metrics, including precision, recall, and F1-score, along with confusion matrices, to evaluate the classification behavior. The results indicate that the One-Class SVM achieved the best overall performance, with the highest recall (99.06%) and F1-score (0.8511), making it highly effective in detecting a broad range of attack types while maintaining a manageable false-positive rate. While Isolation Forest achieved strong precision (78.56%), it underperformed in recall, making it more suitable for applications where false positives must be minimized. LOF delivered a balanced but less robust performance owing to its higher false-alarm rate.

Anahtar Kelimeler

• Anomaly detection
• Unsupervised learning
• Cybersecurity
• One-Class SVM
• Local outlier factor (LOF)

Anomali Tabanlı Saldırı Tespiti için Normal Trafikten Öğrenme: Isolation Forest, LOF ve One-Class SVM Yaklaşımları

Öz

Bu çalışma, ağ tabanlı saldırıların tespitine yönelik olarak yaygın biçimde kullanılan üç gözetimsiz anomali tespit algoritmasının — Isolation Forest, Yerel Aykırı Değer Faktörü (LOF) ve Tek Sınıf Destek Vektör Makinesi (One-Class SVM) — karşılaştırmalı bir analizini sunmaktadır. Çalışmada, kamuya açık bir siber güvenlik veri seti kullanılmış ve hesaplama maliyetini azaltmak ile model performansını artırmak amacıyla Temel Bileşenler Analizi (PCA) uygulanmıştır. Modeller yalnızca normal ağ trafiği verileriyle eğitilmiş, ardından hem normal hem de saldırı örneklerini içeren karışık veri üzerinde test edilmiştir. Performans değerlendirmesi, sınıflandırma başarımını ölçmek amacıyla doğruluk, geri çağırma, F1 puanı ve karışıklık matrisleri gibi temel metrikler kullanılarak gerçekleştirilmiştir. Elde edilen sonuçlar, One-Class SVM algoritmasının %99,06 geri çağırma oranı ve 0,8511 F1 puanı ile en yüksek genel performansı sağladığını, geniş yelpazedeki saldırı türlerini etkili şekilde tespit ederken kabul edilebilir düzeyde yanlış pozitif oranını koruduğunu göstermektedir. Isolation Forest algoritması yüksek doğruluk (%78,56) elde etmiş olmasına rağmen, düşük geri çağırma performansı nedeniyle yanlış pozitiflerin en aza indirilmesinin öncelikli olduğu senaryolarda daha uygun bir seçenek olarak değerlendirilmektedir. LOF algoritması ise görece yüksek yanlış alarm oranı nedeniyle daha dengeli fakat daha az sağlam bir performans sergilemiştir

Anahtar Kelimeler

• Anomali algılama
• Siber güvenlik
• Gözetimsiz öğrenme
• Tek sınıflı svm
• İzolasyon rmanı
• Yerel yykırı değer faktörü

Kaynakça

1. [1] Tatineni, S. 2021. Machine learning approaches for anomaly detection in cybersecurity: a comparative analysis. International Journal of Computer Engineering and Technology, 12(1), 42–50.
2. [2] Segurola-Gil, L., Moreno-Moreno, M., Irigoien, I. ve diğerleri. 2024. Unsupervised anomaly detection approach for cyberattack identification. International Journal of Machine Learning and Cybernetics, 15, 5291–5302.
3. [3] Chandola, V., Banerjee, A., Kumar, V. 2009. Anomaly detection: A survey. ACM Computing Surveys, 41(3), 1–58. https://doi.org/10.1145/1541880.1541882
4. [4] Liu, F. T., Ting, K. M., Zhou, Z. H. 2012. Isolation-based anomaly detection. ACM Transactions on Knowledge Discovery from Data, 6(1), 1–39.
5. [5] Breunig, M. M., Kriegel, H.-P., Ng, R. T., Sander, J. 2000. LOF: Identifying density-based local outliers. ACM SIGMOD Record, 29(2), 93–104.
6. [6] Schölkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., Williamson, R. C. 2001. Estimating the support of a high-dimensional distribution. Neural Computation, 13(7), 1443–1471.
7. [7] Handa, A., Sharma, A., Shukla, S. K. 2019. Machine learning in cybersecurity: a review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(4), e1306.
8. [8] Adiban, M., Siniscalchi, S. M., Salvi, G. 2023. A step-by-step training method for multi-generator GANs with application to anomaly detection and cybersecurity. Neurocomputing, 537, 296–308.

9. [9] Goswami, M. 2024. AI-based anomaly detection for real-time cybersecurity. International Journal of Research and Review in Technology, 3(1), 45–53.
10. [10] Yaseen, A. 2023. The role of machine learning in network anomaly detection for cybersecurity. Sage Scientific Review of Applied Machine Learning, 6(8), 16–34.
11. [11] Alabadi, M., Çelik, Y. 2020. Anomaly detection for cyber-security based on convolution neural network: A survey. Uluslararası İnsan Bilgisayar Etkileşimi, Optimizasyon ve Robotik Uygulamaları Kongresi (HORA), IEEE, 1–14.
12. [12] Inuwa, M. M., Das, R. 2024. A comparative analysis of various machine learning methods for anomaly detection in cyber attacks on IoT networks. Internet of Things, 26, 101162.
13. [13] Hong, J., Liu, C. C., Govindarasu, M. 2014. Integrated anomaly detection for cyber security of the substations. IEEE Transactions on Smart Grid, 5(4), 1643–1653.
14. [14] Choppadandi, A., Kaur, J., Chenchala, P. K., Agarwal, A., Nakra, V., Pandian, P. K. G. 2021. Anomaly detection in cybersecurity: leveraging machine learning algorithms. ESP Journal of Engineering and Technology Advances, 1(2), 34–41.
15. [15] H. Kamal, M. Mashaly, “AE-DTNN: Autoencoder–Dense–Transformer Neural Network Model for Efficient Anomaly-Based Intrusion Detection Systems,” Machine Learning and Knowledge Extraction, vol. 7, no. 3, p. 78, 2025.
16. [16] N. Borgioli, F. Aromolo, L. T. X. Phan, G. Buttazzo, “A convolutional autoencoder architecture for robust network intrusion detection in embedded systems,” Journal of Systems Architecture, vol. 156, p. 103283, 2024.
17. [17] Jia, W., Sun, M., Lian, J. ve diğerleri. 2022. Feature dimensionality reduction: a review. Complex & Intelligent Systems, 8, 2663–2693.
18. [18] Liu, F. T., Ting, K. M., Zhou, Z. H. 2008. Isolation Forest. 2008 IEEE International Conference on Data Mining (ICDM), IEEE, Pisa, Italy, 15–19 Aralık 2008, 413–422.
19. [19] Jolliffe, I. T., Cadima, J. 2016. Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065), 20150202.
20. [20] Scikit-learn developers. sklearn.neighbors.LocalOutlierFactor. https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.LocalOutlierFactor.html (access date: 20.04.2025).
21. [21] Tax, D. M. J., Duin, R. P. W. 2004. Support vector data description. Machine Learning, 54(1), 45–66.
22. [22] Scikit-learn developers. sklearn.svm.OneClassSVM. https://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html (access date: 20.04.2025).
23. [23] Al Farizi, W. S., Hidayah, I., & Rizal, M. N. (2021, September). Isolation forest based anomaly detection: A systematic literature review. In 2021 8th International Conference on Information Technology, Computer and Electrical Engineering (ICITACEE) (pp. 118-122). IEEE.
24. [24] Saadah, B. (2025). ANOMALY DETECTION IN MNIST DATASET USING ONE-CLASS SVM. Jurnal Kecerdasan Buatan dan Teknologi Informasi, 4(3), 264-270.
25. [25] Jolliffe, I. (2011). Principal component analysis. In International encyclopedia of statistical science (pp. 1094-1096). Springer, Berlin, Heidelberg.
26. [26] Almaiah, M. A., Almomani, O., Alsaaidah, A., Al-Otaibi, S., Bani-Hani, N., Hwaitat, A. K. A., ... & Aldhyani, T. H. (2022). Performance investigation of principal component analysis for intrusion detection system using different support vector machine kernels. Electronics, 11(21), 3571.
27. [27] Aoufi, S., Derhab, A., & Guerroumi, M. (2020). Survey of false data injection in smart power grid: Attacks, countermeasures and challenges. Journal of Information Security and Applications, 54, 102518.
28. [28] Ferrag, M.A., Maglaras, L., Moschoyiannis, S., & Janicke, H. (2020). Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study. Journal of Information Security and Applications, 50, 102419.
29. [29] Shone, N., Ngoc, T.N., Phai, V.D., & Shi, Q. (2018). A deep learning approach to network intrusion detection. IEEE Transactions on Emerging Topics in Computational Intelligence, 2(1), 41–50.
30. [30] Vinayakumar, Vinayakumar, R., Alazab, M., Soman, K.P., Poornachandran, P., & Venkatraman, S. (2019). Deep learning approach for intelligent intrusion detection system. IEEE Access, 7, 41525–41550.