Makine Öğrenmesi Modellerinde Spam Tespiti için Hiperparametre Ayarlama Tekniklerinin Performans Değerlendirmesi

Abstract

Hiperparametre seçimi, makine öğrenmesi modellerinin performansını optimize etmede kritik bir rol oynar. Özellikle spam tespiti gibi doğruluk ve hesaplama verimliliğinin önemli olduğu görevlerde, doğru hiperparametre ayarlama tekniklerinin seçimi büyük fark yaratabilir. Bu çalışmada, UCI Makine Öğrenmesi Deposunda yer alan Spambase veri kümesi kullanılarak Altı Makine Öğrenmesi Modeli değerlendirildi: Çok Katmanlı Algılayıcı (MLP), Light Gradient Boosting Machine (LightGBM), Rastgele Orman (RF), Aşırı Gradient Artırma (XGBoost), Karar Ağacı (DT) ve K-En Yakın Komşu (KNN). Bu modellerin optimizasyonu için altı hiperparametre ayarlama yöntemi kullanıldı: Kovaryans Matris Adaptasyonlu Evrim Stratejisi (CMA-ES), Diferansiyel Evrim (DE), Bayesci Optimizasyon (BO), Genetik Algoritma (GA), Ağaç Yapılı Parzen Kestiricisi (TPE) ve Parçacık Sürü Optimizasyonu (PSO). Analiz sonuçları, model ve optimizasyon yöntemi seçiminin tahmin performansı ve kaynak verimliliği üzerinde önemli bir etkisi olduğunu göstermektedir. Deneysel bulgulara göre, XGBoost modeli %98.24 doğruluk oranı ile en yüksek başarıyı elde ederek spam tespiti için etkin bir seçenek olduğunu kanıtlamıştır. LightGBM, %96.74 doğruluk oranı ve yüksek optimizasyon hızıyla performans ve verimlilik arasında dengeli bir alternatif sunmuştur. Karar Ağacı (DT) modelleri, özellikle TPE yöntemiyle sadece 2.81 saniyede optimize edilerek hesaplama açısından en verimli model olarak öne çıkmıştır. Hiperparametre optimizasyon teknikleri açısından bakıldığında, Bayesci Optimizasyon (BO) ve TPE, en verimli yöntemler olarak öne çıkmış ve en düşük zaman maliyetiyle rekabetçi doğruluk seviyeleri elde etmiştir.

Keywords

• Hiperparametre optimizasyonu
• Makine öğrenmesi
• Spam tespiti
• Optimizasyon teknikleri

Performance Evalution of Hyperparameter Tuning Techniques for Machine Learning Models in Spam Detection

Abstract

Hyperparameter selection plays a pivotal role in optimizing the performance of machine learning models, particularly for tasks such as spam detection, where both accuracy and computational efficiency are critical. In this study, the Spambase dataset from the UCI Machine Learning Repository was used to evaluate six machine learning models: Multi-Layer Perceptron (MLP), Light Gradient Boosting Machine (LightGBM), Random Forest (RF), Extreme Gradient Boosting (XGBoost), Decision Tree (DT), and K-Nearest Neighbors (KNN). These models were optimized using six hyperparameter optimization techniques: Covariance Matrix Adaptation Evolution Strategy (CMA-ES), Differential Evolution (DE), Bayesian Optimization (BO), Genetic Algorithm (GA), Tree-structured Parzen Estimator (TPE), and Particle Swarm Optimization (PSO). The analysis highlights the significant impact of model and optimization method selection on predictive performance and resource efficiency. Based on the experimental results, XGBoost achieved the highest accuracy (0.9824), showcasing its effectiveness in spam detection tasks. LightGBM demonstrated a favorable balance between accuracy (0.9674) and optimization speed, making it a practical alternative. Decision Tree models were notable for their computational efficiency, optimizing in as little as 2.81 seconds with TPE. Bayesian Optimization and TPE emerged as the most efficient hyperparameter tuning methods, achieving competitive accuracy with minimal time costs. Future studies could focus on addressing challenges such as computational complexity and evolving spam patterns by exploring advanced optimization strategies and adaptive deep learning models.

Keywords

• Hyperparameter optimization
• Machine learning
• Spam detection
• Optimization techniques

References

1. [1] A. Petrosyan, ‘Global spam volume as percentage of total e-mail traffic from 2011 to 2023’, Statista., 2024.
2. [2] Z. Wang, K. W. Fok, and V. L. Thing, ‘Machine learning for encrypted malicious traffic detection: Approaches, datasets and comparative study’, Computers & Security, vol. 113, p. 102542, 2022.
3. [3] R. Agarwal et al., ‘A novel approach for spam detection using natural language processing with AMALS models’, IEEE Access, vol. 12, pp. 124298–124313, 2024.
4. [4] L. Yang and A. Shami, ‘On hyperparameter optimization of machine learning algorithms: Theory and practice’, Neurocomputing, vol. 415, pp. 295–316, 2020.
5. [5] Y. A. Ali, E. M. Awwad, M. Al-Razgan, and A. Maarouf, ‘Hyperparameter search for machine learning algorithms for optimizing the computational complexity’, Processes, vol. 11, no. 2, p. 349, 2023.
6. [6] J. Wu, X.-Y. Chen, H. Zhang, L.-D. Xiong, H. Lei, and S.-H. Deng, ‘Hyperparameter optimization for machine learning models based on Bayesian optimization’, Journal of Electronic Science and Technology, vol. 17, no. 1, pp. 26–40, 2019.
7. [7] J. Guo et al., ‘Prediction of heating and cooling loads based on light gradient boosting machine algorithms’, Building and Environment, vol. 236, p. 110252, 2023.
8. [8] E. Elgeldawi, A. Sayed, A. R. Galal, and A. M. Zaki, ‘Hyperparameter tuning for machine learning algorithms used for arabic sentiment analysis’, presented at the Informatics, MDPI, 2021, p. 79.

9. [9] T. O. Omotehinwa and D. O. Oyewola, ‘Hyperparameter optimization of ensemble models for spam email detection’, Applied Sciences, vol. 13, no. 3, p. 1971, 2023.
10. [10] Y. Kontsewaya, E. Antonov, and A. Artamonov, ‘Evaluating the effectiveness of machine learning methods for spam detection’, Procedia Computer Science, vol. 190, pp. 479–486, 2021.
11. [11] M. Açikkar, ‘Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression’, Turkish Journal of Electrical Engineering and Computer Sciences, vol. 32, no. 1, pp. 68–92, 2024.
12. [12] D. Budiman, Z. Zayyan, A. Mardiana, and A. A. Mahrani, ‘Email spam detection: a comparison of svm and naive bayes using bayesian optimization and grid search parameters’, Journal of Student Research Exploration, vol. 2, no. 1, pp. 53–64, 2024.
13. [13] T. T. Khoei, S. Ismail, and N. Kaabouch, ‘Boosting-based models with tree-structured parzen estimator optimization to detect intrusion attacks on smart grid’, presented at the 2021 IEEE 12th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), IEEE, 2021, pp. 0165–0170.
14. [14] M. M. Belal and D. M. Sundaram, ‘An intelligent protection framework for intrusion detection in cloud environment based on covariance matrix self-adaptation evolution strategy and multi-criteria decision-making’, Journal of Intelligent & Fuzzy Systems, vol. 44, no. 6, pp. 8971–9001, 2023.
15. [15] A. Sinhmar, V. Malhotra, R. Yadav, and M. Kumar, ‘Spam detection using genetic algorithm optimized LSTM model’, presented at the Computer Networks and Inventive Communication Technologies: Proceedings of Fourth ICCNCT 2021, Springer, 2022, pp. 59–72.
16. [16] S. Kumar Birthriya, P. Ahlawat, and A. Kumar Jain, ‘An efficient spam and phishing Email filtering approach using deep learning and bio-inspired particle swarm optimization’, International Journal of Computing and Digital Systems, vol. 15, no. 1, pp. 1–11, 2024.
17. [17] M. Hopkins, E. Reeber, G. Forman, and J. Suermondt, ‘Spambase. UCI Machine Learning Repository’, 1999.
18. [18] J. Jin, C. Yang, and Y. Zhang, ‘An improved CMA-ES for solving large scale optimization problem’, presented at the Advances in Swarm Intelligence: 11th International Conference, ICSI 2020, Belgrade, Serbia, July 14–20, 2020, Proceedings 11, Springer, 2020, pp. 386–396.
19. [19] M. Yeo, B. Kim, H. Kim, and Y. Jeong, ‘Channel phase extraction for a coherent beam combining system using a 2D target intensity image and the CMA-ES algorithm’, Journal of the Korean Physical Society, vol. 85, no. 2, pp. 120–128, 2024.
20. [20] B. Ghahremani, M. Bitaraf, and H. Rahami, ‘A fast-convergent approach for damage assessment using CMA-ES optimization algorithm and modal parameters’, Journal of Civil Structural Health Monitoring, vol. 10, pp. 497–511, 2020.
21. [21] M. Nomura, S. Watanabe, Y. Akimoto, Y. Ozaki, and M. Onishi, ‘Warm starting CMA-ES for hyperparameter optimization’, presented at theProceedings of the AAAI conference on artificial intelligence, pp. 9188–9196, 2021.
22. [22] I. Loshchilov and F. Hutter, ‘CMA-ES for hyperparameter optimization of deep neural networks’, arXiv preprint arXiv:1604.07269, 2016.
23. [23] T. Eltaeib and A. Mahmood, ‘Differential evolution: A survey and analysis’, Applied Sciences, vol. 8, no. 10, p. 1945, 2018.
24. [24] Z. Meng and C. Yang, ‘Hip-DE: Historical population based mutation strategy in differential evolution with parameter adaptive mechanism’, Information Sciences, vol. 562, pp. 44–77, 2021.
25. [25] W. Deng, S. Shang, X. Cai, H. Zhao, Y. Song, and J. Xu, ‘An improved differential evolution algorithm and its application in optimization problem’, Soft Computing, vol. 25, pp. 5277–5298, 2021.
26. [26] B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. De Freitas, ‘Taking the human out of the loop: A review of Bayesian optimization’, Proceedings of the IEEE, vol. 104, no. 1, pp. 148–175, 2015.
27. [27] M. Zulfiqar, K. A. Gamage, M. Kamran, and M. B. Rasheed, ‘Hyperparameter optimization of bayesian neural network using bayesian optimization and intelligent feature engineering for load forecasting’, Sensors, vol. 22, no. 12, p. 4446, 2022.
28. [28] J. Snoek, H. Larochelle, and R. P. Adams, ‘Practical bayesian optimization of machine learning algorithms’, Advances in neural information processing systems, vol. 25, 2012.
29. [29] A. H. Victoria and G. Maragatham, ‘Automatic tuning of hyperparameters using Bayesian optimization’, Evolving Systems, vol. 12, no. 1, pp. 217–223, 2021.
30. [30] M. A. Albadr, S. Tiun, M. Ayob, and F. Al-Dhief, ‘Genetic algorithm based on natural selection theory for optimization problems’, Symmetry, vol. 12, no. 11, p. 1758, 2020.
31. [31] S. Katoch, S. S. Chauhan, and V. Kumar, ‘A review on genetic algorithm: past, present, and future’, Multimedia tools and applications, vol. 80, pp. 8091–8126, 2021.
32. [32] S. Lee, J. Kim, H. Kang, D.-Y. Kang, and J. Park, ‘Genetic algorithm based deep learning neural network structure and hyperparameter optimization’, Applied Sciences, vol. 11, no. 2, p. 744, 2021.
33. [33] L. Tani, D. Rand, C. Veelken, and M. Kadastik, ‘Evolutionary algorithms for hyperparameter optimization in machine learning for application in high energy physics’, The European Physical Journal C, vol. 81, pp. 1–9, 2021.
34. [34] M. Rajalakshmi and V. Sulochana, ‘Enhancing deep learning model performance in air quality classification through probabilistic hyperparameter tuning with tree-structured Parzen estimators’, The Scientific Temper, vol. 14, no. 04, pp. 1244–1250, 2023.
35. [35] G. Rong et al., ‘Comparison of tree-structured parzen estimator optimization in three typical neural network models for landslide susceptibility assessment’, Remote Sensing, vol. 13, no. 22, p. 4694, 2021.
36. [36] R. Islam, A. Sultana, and M. N. Tuhin, ‘A comparative analysis of machine learning algorithms with tree-structured parzen estimator for liver disease prediction’, Healthcare Analytics, vol. 6, p. 100358, 2024.
37. [37] S. Watanabe, ‘Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance’, arXiv preprint arXiv:2304.11127, 2023.
38. [38] L. A. Demidova and A. V. Filatov, ‘Optimization of hyperparameters with constraints on time and memory for the classification model of the hard drives states’, presented at the 2022 International Conference on Information Technologies (InfoTech), IEEE, 2022, pp. 1–4.
39. [39] D. D. Ramírez-Ochoa, L. A. Pérez-Domínguez, E.-A. Martínez-Gómez, and D. Luviano-Cruz, ‘PSO, a swarm intelligence-based evolutionary algorithm as a decision-making strategy: A review’, Symmetry, vol. 14, no. 3, p. 455, 2022.
40. [40] M. F. A. Foysal, N. Sultana, T. A. Rimi, and M. H. Rifat, ‘Convolutional neural network hyper-parameter optimization using particle swarm optimization’, presented at the Emerging Technologies in Data Mining and Information Security: Proceedings of IEMIS 2020, Volume 2, Springer, 2021, pp. 363–373.
41. [41] Z. Fouad, M. Alfonse, M. Roushdy, and A.-B. M. Salem, ‘Hyper-parameter optimization of convolutional neural network based on particle swarm optimization algorithm’, Bulletin of Electrical Engineering and Informatics, vol. 10, no. 6, pp. 3377–3384, 2021.
42. [42] Y. Guo, J.-Y. Li, and Z.-H. Zhan, ‘Efficient hyperparameter optimization for convolution neural networks in deep learning: A distributed particle swarm optimization approach’, Cybernetics and Systems, vol. 52, no. 1, pp. 36–57, 2020.
43. [43] O. Kramer and O. Kramer, ‘K-nearest neighbors’, Dimensionality reduction with unsupervised nearest neighbors, pp. 13–23, 2013.
44. [44] R. K. Halder, M. N. Uddin, M. A. Uddin, S. Aryal, and A. Khraisat, ‘Enhancing K-nearest neighbor algorithm: a comprehensive review and performance analysis of modifications’, Journal of Big Data, vol. 11, no. 1, p. 113, 2024.
45. [45] S. R. Safavian and D. Landgrebe, ‘A survey of decision tree classifier methodology’, IEEE transactions on systems, man, and cybernetics, vol. 21, no. 3, pp. 660–674, 1991.
46. [46] B. Charbuty and A. Abdulazeez, ‘Classification based on decision tree algorithm for machine learning’, Journal of Applied Science and Technology Trends, vol. 2, no. 01, pp. 20–28, 2021.
47. [47] Y. Liu, Y. Wang, and J. Zhang, ‘New machine learning algorithm: Random forest’, in Information Computing and Applications: Third International Conference, Chengde, 2012, pp. 246–252.
48. [48] A. A. Akinyelu and A. O. Adewumi, ‘Classification of phishing email using random forest machine learning technique’, Journal of Applied Mathematics, vol. 2014, no. 1, p. 425731, 2014.
49. [49] G. Coffie and S. Cudjoe, ‘Using extreme gradient boosting (XGBoost) machine learning to predict construction cost overruns’, International Journal of Construction Management, vol. 24, no. 16, pp. 1742–1750, 2024.
50. [50] T. Chen and C. Guestrin, ‘Xgboost: A scalable tree boosting system’, presented at the Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785–794.
51. [51] F. Alzamzami, M. Hoda, and A. El Saddik, ‘Light gradient boosting machine for general sentiment classification on short texts: a comparative evaluation’, IEEE access, vol. 8, pp. 101840–101858, 2020.
52. [52] J. Fan, X. Ma, L. Wu, F. Zhang, X. Yu, and W. Zeng, ‘Light Gradient Boosting Machine: An efficient soft computing model for estimating daily reference evapotranspiration with local and external meteorological data’, Agricultural water management, vol. 225, p. 105758, 2019.
53. [53] R. Kruse, S. Mostaghim, C. Borgelt, C. Braune, and M. Steinbrecher, ‘Multi-layer perceptrons’, in Computational intelligence: a methodological introduction, Springer, 2022, pp. 53–124.
54. [54] M. Aitkin and R. Foxall, ‘Statistical modelling of artificial neural networks using the multi-layer perceptron’, Statistics and Computing, vol. 13, pp. 227–239, 2003.