The Analysis of a Linear Classifier Developed through Particle Swarm Optimization

Abstract

Meta-heuristics are high-level approaches developed to discover a heuristic that provides a reasonable solution to many varieties of optimization problems. The classification problems contain a sort of optimization problem. Simply, the objective herein is to reduce the number of misclassified instances. In this paper, the question of whether meta-heuristic methods can be used to construct linear models or not is answered. To this end, Particle Swarm Optimization (PSO) has been engaged to address linear classification problems. The Particle Swarm Classifier (PSC) with a certain objective function has been compared with Support Vector Machine (SVM), Perceptron Learning Rule (PLR), and Logistic Regression (LR) applied to fifteen data sets. The experimental results point out that PSC can compete with the other classifiers, and it turns out to be superior to other classifiers for some binary classification problems. Furthermore, the average classification accuracies of PSC, SVM, LR, and PLR are 80.8%, 80.6%, 80.9%, and 57.7%, respectively. In order to enhance the classification performance of PSC, more advanced objective functions can be developed. Further, the classification accuracy can be boosted more by constructing tighter constraints via another meta-heuristic.

Keywords

• machine learning
• artificial intelligence
• particle swarm optimization
• meta-heuristic algorithms
• supervised learning

Parçacık Sürü Optimizasyonu Yoluyla Geliştirilen Doğrusal Bir Sınıflandırıcının Analizi

Abstract

Meta-sezgisel yöntemler, çok çeşitli optimizasyon problemlerine uygun bir çözüm sağlayan sezgisel bir yöntemi keşfetmek için geliştirilmiş üst düzey yaklaşımlardır. Sınıflandırma problemleri bir tür optimizasyon problemi içerir. Kısacası, sınıflandırma problemlerinde amaç yanlış sınıflandırılan örneklerin sayısını azaltmaktır. Bu makalede, meta sezgisel yöntemlerin doğrusal modeller oluşturmak için kullanılıp kullanılamayacağı sorusunu cevaplamaktır. Bu amaçla, doğrusal sınıflandırma problemlerini çözmek için Parçacık Sürü Optimizasyonu (PSO) devreye alınmıştır. Belirli bir amaç fonksiyonuna sahip Parçacık Sürü Sınıflandırıcısı (PSC), on beş veri kümesi üzerine uygulanan Destek Vektör Makinesi (SVM), Perceptron Learning Rule (PLR) ve Logistic Regresyon (LR) ile karşılaştırılmıştır. Deneysel sonuçlar, PSC'nin diğer sınıflandırıcılarla rekabet edebildiğini ve bazı ikili sınıflandırma problemlerinde diğer sınıflandırıcılardan üstün olduğunu göstermektedir. Ayrıca, PSC, SVM, LR ve PLR'nin ortalama sınıflandırma doğrulukları sırasıyla %80,8, %80,6, %80,9 ve %57,7'dir. PSC'nin sınıflandırma performansını artırmak için daha gelişmiş amaç fonksiyonları geliştirilebilir. Ayrıca, başka bir meta sezgisel yöntemle daha sıkı kısıtlamalar oluşturarak sınıflandırma doğruluğu daha fazla artırılabilir.

Keywords

• makine öğrenmesi
• yapay zeka
• parçacık sürü optimizasyonu
• meta sezgisel algoritmalar
• denetimli öğrenme

References

1. Ahmad, Z., Li, J., & Mahmood, T. (2023). Adaptive Hyperparameter Fine-Tuning for Boosting the Robustness and Quality of the Particle Swarm Optimization Algorithm for Non-Linear RBF Neural Network Modelling and Its Applications. Mathematics, 11(1), 242. https://doi.org/10.3390/math11010242
2. Almasi, O. N., & Khooban, M. H. (2018). A parsimonious SVM model selection criterion for classification of real-world data sets via an adaptive population-based algorithm. Neural Computing and Applications, 30(11), 3421–3429. https://doi.org/10.1007/s00521-017-2930-y
3. Asif, M., Nagra, A. A., Ahmad, M. Bin, & Masood, K. (2022). Feature Selection Empowered by Self-Inertia Weight Adaptive Particle Swarm Optimization for Text Classification. Applied Artificial Intelligence, 36(1). https://doi.org/10.1080/08839514.2021.2004345
4. Athira Lekshmi, B. A., Linsely, J. A., Queen, M. . F., & Babu Aurtherson, P. (2018). Feature Extraction and Image Classification Using Particle Swarm Optimization by Evolving Rotation-Invariant Image Descriptors. 2018 International Conference on Emerging Trends and Innovations In Engineering And Technological Research (ICETIETR), 1–5. https://doi.org/10.1109/ICETIETR.2018.8529083
5. Ay, Ş., Ekinci, E., & Garip, Z. (2023). A comparative analysis of meta-heuristic optimization algorithms for feature selection on ML-based classification of heart-related diseases. The Journal of Supercomputing. https://doi.org/10.1007/s11227-023-05132-3
6. Aydin, F. (2023). Unsupervised instance selection via conjectural hyperrectangles. Neural Computing and Applications, 35(7), 5335–5349. https://doi.org/10.1007/s00521-022-07974-z
7. Aziz, Y., & Memon, K. H. (2023). Fast geometrical extraction of nearest neighbors from multi-dimensional data. Pattern Recognition, 136, 109183. https://doi.org/10.1016/j.patcog.2022.109183
8. Balamurugan, R., Natarajan, A. M., & Premalatha, K. (2015). Stellar-Mass Black Hole Optimization for Biclustering Microarray Gene Expression Data. Applied Artificial Intelligence, 29(4), 353–381. https://doi.org/10.1080/08839514.2015.1016391

9. Baygin, N., Baygin, M., & Karakose, M. (2019). A SVM-PSO Classifier for Robot Motion in Environment with Obstacles. 2019 International Artificial Intelligence and Data Processing Symposium (IDAP), 1–5. https://doi.org/10.1109/IDAP.2019.8875921
10. Berkson, J. (1944). Application of the Logistic Function to Bio-Assay. Journal of the American Statistical Association, 39(227), 357–365. https://doi.org/10.1080/01621459.1944.10500699
11. Bratton, D., & Kennedy, J. (2007). Defining a Standard for Particle Swarm Optimization. 2007 IEEE Swarm Intelligence Symposium, 120–127. https://doi.org/10.1109/SIS.2007.368035
12. Bumin, M., & Ozcalici, M. (2023). Predicting the direction of financial dollarization movement with genetic algorithm and machine learning algorithms: The case of Turkey. Expert Systems with Applications, 213, 119301. https://doi.org/10.1016/j.eswa.2022.119301
13. Chamkalani, A., Zendehboudi, S., Bahadori, A., Kharrat, R., Chamkalani, R., James, L., & Chatzis, I. (2014). Integration of LSSVM technique with PSO to determine asphaltene deposition. Journal of Petroleum Science and Engineering, 124, 243–253. https://doi.org/10.1016/j.petrol.2014.10.001
14. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018
15. De Falco, I., Della Cioppa, A., & Tarantino, E. (2007). Facing classification problems with Particle Swarm Optimization. Applied Soft Computing, 7(3), 652–658. https://doi.org/10.1016/j.asoc.2005.09.004
16. Elshamy, W., Emara, H. M., & Bahgat, A. (2007). Clubs-based Particle Swarm Optimization. 2007 IEEE Swarm Intelligence Symposium, 289–296. https://doi.org/10.1109/SIS.2007.367950
17. Eshtay, M., Faris, H., Heidari, A. A., Al-Zoubi, A. M., & Aljarah, I. (2021). AutoRWN: automatic construction and training of random weight networks using competitive swarm of agents. Neural Computing and Applications, 33(11), 5507–5524. https://doi.org/10.1007/s00521-020-05329-0
18. Hu, J., Ou, X., Liang, P., & Li, B. (2022). Applying particle swarm optimization-based decision tree classifier for wart treatment selection. Complex & Intelligent Systems, 8(1), 163–177. https://doi.org/10.1007/s40747-021-00348-3
19. Karakurt, M., Oymak, E. A., Hark, H., Erdoğan, M. C., & Karcı, A. (2022). Karcı Sinir Ağlarının Uygulaması ve Performans Analizi. Computer Science, 7(2), 68–80. https://doi.org/10.53070/bbd.1194017
20. Kennedy, J., & Eberhart, R. (1995). Particle swarm optimization. Proceedings of ICNN’95 - International Conference on Neural Networks, 4, 1942–1948. https://doi.org/10.1109/ICNN.1995.488968
21. Khennak, I., Drias, H., Drias, Y., Bendakir, F., & Hamdi, S. (2023). I/F-Race tuned firefly algorithm and particle swarm optimization for K-medoids-based clustering. Evolutionary Intelligence, 16(1), 351–373. https://doi.org/10.1007/s12065-022-00794-z
22. Kotary, D. K., & Nanda, S. J. (2020). Distributed robust data clustering in wireless sensor networks using diffusion moth flame optimization. Engineering Applications of Artificial Intelligence, 87, 103342. https://doi.org/10.1016/j.engappai.2019.103342
23. Lai, D. T. C., & Sato, Y. (2021). An Empirical Study of Cluster-Based MOEA/D Bare Bones PSO for Data Clustering †. Algorithms, 14(11), 338. https://doi.org/10.3390/a14110338
24. Lawrence, T., Zhang, L., Rogage, K., & Lim, C. P. (2021). Evolving Deep Architecture Generation with Residual Connections for Image Classification Using Particle Swarm Optimization. Sensors, 21(23), 7936. https://doi.org/10.3390/s21237936
25. Lenat, D. B., & Feigenbaum, E. A. (1991). On the thresholds of knowledge. Artificial Intelligence, 47(1–3), 185–250. https://doi.org/10.1016/0004-3702(91)90055-O
26. Ma, T., Wang, C., Wang, J., Cheng, J., & Chen, X. (2019). Particle-swarm optimization of ensemble neural networks with negative correlation learning for forecasting short-term wind speed of wind farms in western China. Information Sciences, 505, 157–182. https://doi.org/10.1016/j.ins.2019.07.074
27. Mason, K., Duggan, J., & Howley, E. (2018). A meta optimisation analysis of particle swarm optimisation velocity update equations for watershed management learning. Applied Soft Computing, 62, 148–161. https://doi.org/10.1016/j.asoc.2017.10.018
28. Mezura-Montes, E., & Coello Coello, C. A. (2011). Constraint-handling in nature-inspired numerical optimization: Past, present and future. Swarm and Evolutionary Computation, 1(4), 173–194. https://doi.org/10.1016/j.swevo.2011.10.001
29. Oliveira, M., Pinheiro, D., Andrade, B., Bastos-Filho, C., & Menezes, R. (2016). Communication Diversity in Particle Swarm Optimizers. In Lecture Notes in Computer Science (pp. 77–88). Springer, Cham. https://doi.org/10.1007/978-3-319-44427-7_7
30. Omran, M. G., Engelbrecht, A. P., & Salman, A. (2002). Image Classification Using Particle Swarm Optimization. The Fourth Asia–Pacific Conference on Recent Advances in Simulated Evolution and Learning, 347–365. https://doi.org/10.1142/9789812561794_0019
31. Omran, M. G. H., Engelbrecht, A. P., & Salman, A. (2006). Particle Swarm Optimization for Pattern Recognition and Image Processing. In A. Abraham, C. Grosan, & V. Ramos (Eds.), Swarm Intelligence in Data Mining (pp. 125–151). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-34956-3_6
32. Pedersen, M. E. H. (2010). Good parameters for particle swarm optimization. https://citeseerx.ist.psu.edu/doc/10.1.1.298.4359
33. Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386–408. https://doi.org/10.1037/h0042519
34. Russell, S. J., & Norvig, P. (2010). Artificial Intelligence: A Modern Approach (3rd ed.). Prentice Hall.
35. Sivrikaya, Ö. E., Yüksekgönül, M., & Baydoğan, M. G. (2021). Learning prototypes for multiple instance learning. Turkish Journal of Electrical Engineering & Computer Sciences, 29(7), 2901–2919. https://doi.org/10.3906/elk-2101-103
36. Sousa, T., Silva, A., & Neves, A. (2004). Particle Swarm based Data Mining Algorithms for classification tasks. Parallel Computing, 30(5–6), 767–783. https://doi.org/10.1016/j.parco.2003.12.015
37. Taherkhani, M., & Safabakhsh, R. (2016). A novel stability-based adaptive inertia weight for particle swarm optimization. Applied Soft Computing, 38, 281–295. https://doi.org/10.1016/j.asoc.2015.10.004
38. Tan, T. Y., Zhang, L., & Lim, C. P. (2019). Intelligent skin cancer diagnosis using improved particle swarm optimization and deep learning models. Applied Soft Computing, 84, 105725. https://doi.org/10.1016/j.asoc.2019.105725
39. Telikani, A., Tahmassebi, A., Banzhaf, W., & Gandomi, A. H. (2022). Evolutionary Machine Learning: A Survey. ACM Computing Surveys, 54(8), 1–35. https://doi.org/10.1145/3467477
40. Yilmaz, A., & Yolcu, U. (2022). Dendritic neuron model neural network trained by modified particle swarm optimization for time‐series forecasting. Journal of Forecasting, 41(4), 793–809. https://doi.org/10.1002/for.2833
41. Yin, P.-Y., Glover, F., Laguna, M., & Zhu, J.-X. (2011). A Complementary Cyber Swarm Algorithm. International Journal of Swarm Intelligence Research, 2(2), 22–41. https://doi.org/10.4018/jsir.2011040102
42. Yuan, G.-X., Ho, C.-H., & Lin, C.-J. (2012). Recent Advances of Large-Scale Linear Classification. Proceedings of the IEEE, 100(9), 2584–2603. https://doi.org/10.1109/JPROC.2012.2188013