The Analysis of a Linear Classifier Developed through Particle Swarm Optimization

Year 2023, , 36 - 49, 08.06.2023

Fatih Aydın

https://doi.org/10.53070/bbd.1259377

Abstract

Meta-heuristics are high-level approaches developed to discover a heuristic that provides a reasonable solution to many varieties of optimization problems. The classification problems contain a sort of optimization problem. Simply, the objective herein is to reduce the number of misclassified instances. In this paper, the question of whether meta-heuristic methods can be used to construct linear models or not is answered. To this end, Particle Swarm Optimization (PSO) has been engaged to address linear classification problems. The Particle Swarm Classifier (PSC) with a certain objective function has been compared with Support Vector Machine (SVM), Perceptron Learning Rule (PLR), and Logistic Regression (LR) applied to fifteen data sets. The experimental results point out that PSC can compete with the other classifiers, and it turns out to be superior to other classifiers for some binary classification problems. Furthermore, the average classification accuracies of PSC, SVM, LR, and PLR are 80.8%, 80.6%, 80.9%, and 57.7%, respectively. In order to enhance the classification performance of PSC, more advanced objective functions can be developed. Further, the classification accuracy can be boosted more by constructing tighter constraints via another meta-heuristic.

Keywords

machine learning, artificial intelligence, particle swarm optimization, meta-heuristic algorithms, supervised learning

Parçacık Sürü Optimizasyonu Yoluyla Geliştirilen Doğrusal Bir Sınıflandırıcının Analizi

Abstract

Meta-sezgisel yöntemler, çok çeşitli optimizasyon problemlerine uygun bir çözüm sağlayan sezgisel bir yöntemi keşfetmek için geliştirilmiş üst düzey yaklaşımlardır. Sınıflandırma problemleri bir tür optimizasyon problemi içerir. Kısacası, sınıflandırma problemlerinde amaç yanlış sınıflandırılan örneklerin sayısını azaltmaktır. Bu makalede, meta sezgisel yöntemlerin doğrusal modeller oluşturmak için kullanılıp kullanılamayacağı sorusunu cevaplamaktır. Bu amaçla, doğrusal sınıflandırma problemlerini çözmek için Parçacık Sürü Optimizasyonu (PSO) devreye alınmıştır. Belirli bir amaç fonksiyonuna sahip Parçacık Sürü Sınıflandırıcısı (PSC), on beş veri kümesi üzerine uygulanan Destek Vektör Makinesi (SVM), Perceptron Learning Rule (PLR) ve Logistic Regresyon (LR) ile karşılaştırılmıştır. Deneysel sonuçlar, PSC'nin diğer sınıflandırıcılarla rekabet edebildiğini ve bazı ikili sınıflandırma problemlerinde diğer sınıflandırıcılardan üstün olduğunu göstermektedir. Ayrıca, PSC, SVM, LR ve PLR'nin ortalama sınıflandırma doğrulukları sırasıyla %80,8, %80,6, %80,9 ve %57,7'dir. PSC'nin sınıflandırma performansını artırmak için daha gelişmiş amaç fonksiyonları geliştirilebilir. Ayrıca, başka bir meta sezgisel yöntemle daha sıkı kısıtlamalar oluşturarak sınıflandırma doğruluğu daha fazla artırılabilir.

Keywords

makine öğrenmesi, yapay zeka, parçacık sürü optimizasyonu, meta sezgisel algoritmalar, denetimli öğrenme

References

• Ahmad, Z., Li, J., & Mahmood, T. (2023). Adaptive Hyperparameter Fine-Tuning for Boosting the Robustness and Quality of the Particle Swarm Optimization Algorithm for Non-Linear RBF Neural Network Modelling and Its Applications. Mathematics, 11(1), 242. https://doi.org/10.3390/math11010242
• Almasi, O. N., & Khooban, M. H. (2018). A parsimonious SVM model selection criterion for classification of real-world data sets via an adaptive population-based algorithm. Neural Computing and Applications, 30(11), 3421–3429. https://doi.org/10.1007/s00521-017-2930-y
• Asif, M., Nagra, A. A., Ahmad, M. Bin, & Masood, K. (2022). Feature Selection Empowered by Self-Inertia Weight Adaptive Particle Swarm Optimization for Text Classification. Applied Artificial Intelligence, 36(1). https://doi.org/10.1080/08839514.2021.2004345
• Athira Lekshmi, B. A., Linsely, J. A., Queen, M. . F., & Babu Aurtherson, P. (2018). Feature Extraction and Image Classification Using Particle Swarm Optimization by Evolving Rotation-Invariant Image Descriptors. 2018 International Conference on Emerging Trends and Innovations In Engineering And Technological Research (ICETIETR), 1–5. https://doi.org/10.1109/ICETIETR.2018.8529083
• Ay, Ş., Ekinci, E., & Garip, Z. (2023). A comparative analysis of meta-heuristic optimization algorithms for feature selection on ML-based classification of heart-related diseases. The Journal of Supercomputing. https://doi.org/10.1007/s11227-023-05132-3
• Aydin, F. (2023). Unsupervised instance selection via conjectural hyperrectangles. Neural Computing and Applications, 35(7), 5335–5349. https://doi.org/10.1007/s00521-022-07974-z
• Aziz, Y., & Memon, K. H. (2023). Fast geometrical extraction of nearest neighbors from multi-dimensional data. Pattern Recognition, 136, 109183. https://doi.org/10.1016/j.patcog.2022.109183
• Balamurugan, R., Natarajan, A. M., & Premalatha, K. (2015). Stellar-Mass Black Hole Optimization for Biclustering Microarray Gene Expression Data. Applied Artificial Intelligence, 29(4), 353–381. https://doi.org/10.1080/08839514.2015.1016391
• Baygin, N., Baygin, M., & Karakose, M. (2019). A SVM-PSO Classifier for Robot Motion in Environment with Obstacles. 2019 International Artificial Intelligence and Data Processing Symposium (IDAP), 1–5. https://doi.org/10.1109/IDAP.2019.8875921
• Berkson, J. (1944). Application of the Logistic Function to Bio-Assay. Journal of the American Statistical Association, 39(227), 357–365. https://doi.org/10.1080/01621459.1944.10500699
• Bratton, D., & Kennedy, J. (2007). Defining a Standard for Particle Swarm Optimization. 2007 IEEE Swarm Intelligence Symposium, 120–127. https://doi.org/10.1109/SIS.2007.368035
• Bumin, M., & Ozcalici, M. (2023). Predicting the direction of financial dollarization movement with genetic algorithm and machine learning algorithms: The case of Turkey. Expert Systems with Applications, 213, 119301. https://doi.org/10.1016/j.eswa.2022.119301
• Chamkalani, A., Zendehboudi, S., Bahadori, A., Kharrat, R., Chamkalani, R., James, L., & Chatzis, I. (2014). Integration of LSSVM technique with PSO to determine asphaltene deposition. Journal of Petroleum Science and Engineering, 124, 243–253. https://doi.org/10.1016/j.petrol.2014.10.001
• Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018
• De Falco, I., Della Cioppa, A., & Tarantino, E. (2007). Facing classification problems with Particle Swarm Optimization. Applied Soft Computing, 7(3), 652–658. https://doi.org/10.1016/j.asoc.2005.09.004
• Elshamy, W., Emara, H. M., & Bahgat, A. (2007). Clubs-based Particle Swarm Optimization. 2007 IEEE Swarm Intelligence Symposium, 289–296. https://doi.org/10.1109/SIS.2007.367950
• Eshtay, M., Faris, H., Heidari, A. A., Al-Zoubi, A. M., & Aljarah, I. (2021). AutoRWN: automatic construction and training of random weight networks using competitive swarm of agents. Neural Computing and Applications, 33(11), 5507–5524. https://doi.org/10.1007/s00521-020-05329-0
• Hu, J., Ou, X., Liang, P., & Li, B. (2022). Applying particle swarm optimization-based decision tree classifier for wart treatment selection. Complex & Intelligent Systems, 8(1), 163–177. https://doi.org/10.1007/s40747-021-00348-3
• Karakurt, M., Oymak, E. A., Hark, H., Erdoğan, M. C., & Karcı, A. (2022). Karcı Sinir Ağlarının Uygulaması ve Performans Analizi. Computer Science, 7(2), 68–80. https://doi.org/10.53070/bbd.1194017
• Kennedy, J., & Eberhart, R. (1995). Particle swarm optimization. Proceedings of ICNN’95 - International Conference on Neural Networks, 4, 1942–1948. https://doi.org/10.1109/ICNN.1995.488968
• Khennak, I., Drias, H., Drias, Y., Bendakir, F., & Hamdi, S. (2023). I/F-Race tuned firefly algorithm and particle swarm optimization for K-medoids-based clustering. Evolutionary Intelligence, 16(1), 351–373. https://doi.org/10.1007/s12065-022-00794-z
• Kotary, D. K., & Nanda, S. J. (2020). Distributed robust data clustering in wireless sensor networks using diffusion moth flame optimization. Engineering Applications of Artificial Intelligence, 87, 103342. https://doi.org/10.1016/j.engappai.2019.103342
• Lai, D. T. C., & Sato, Y. (2021). An Empirical Study of Cluster-Based MOEA/D Bare Bones PSO for Data Clustering †. Algorithms, 14(11), 338. https://doi.org/10.3390/a14110338
• Lawrence, T., Zhang, L., Rogage, K., & Lim, C. P. (2021). Evolving Deep Architecture Generation with Residual Connections for Image Classification Using Particle Swarm Optimization. Sensors, 21(23), 7936. https://doi.org/10.3390/s21237936
• Lenat, D. B., & Feigenbaum, E. A. (1991). On the thresholds of knowledge. Artificial Intelligence, 47(1–3), 185–250. https://doi.org/10.1016/0004-3702(91)90055-O
• Ma, T., Wang, C., Wang, J., Cheng, J., & Chen, X. (2019). Particle-swarm optimization of ensemble neural networks with negative correlation learning for forecasting short-term wind speed of wind farms in western China. Information Sciences, 505, 157–182. https://doi.org/10.1016/j.ins.2019.07.074
• Mason, K., Duggan, J., & Howley, E. (2018). A meta optimisation analysis of particle swarm optimisation velocity update equations for watershed management learning. Applied Soft Computing, 62, 148–161. https://doi.org/10.1016/j.asoc.2017.10.018
• Mezura-Montes, E., & Coello Coello, C. A. (2011). Constraint-handling in nature-inspired numerical optimization: Past, present and future. Swarm and Evolutionary Computation, 1(4), 173–194. https://doi.org/10.1016/j.swevo.2011.10.001
• Oliveira, M., Pinheiro, D., Andrade, B., Bastos-Filho, C., & Menezes, R. (2016). Communication Diversity in Particle Swarm Optimizers. In Lecture Notes in Computer Science (pp. 77–88). Springer, Cham. https://doi.org/10.1007/978-3-319-44427-7_7
• Omran, M. G., Engelbrecht, A. P., & Salman, A. (2002). Image Classification Using Particle Swarm Optimization. The Fourth Asia–Pacific Conference on Recent Advances in Simulated Evolution and Learning, 347–365. https://doi.org/10.1142/9789812561794_0019
• Omran, M. G. H., Engelbrecht, A. P., & Salman, A. (2006). Particle Swarm Optimization for Pattern Recognition and Image Processing. In A. Abraham, C. Grosan, & V. Ramos (Eds.), Swarm Intelligence in Data Mining (pp. 125–151). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-34956-3_6
• Pedersen, M. E. H. (2010). Good parameters for particle swarm optimization. https://citeseerx.ist.psu.edu/doc/10.1.1.298.4359
• Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386–408. https://doi.org/10.1037/h0042519
• Russell, S. J., & Norvig, P. (2010). Artificial Intelligence: A Modern Approach (3rd ed.). Prentice Hall.
• Sivrikaya, Ö. E., Yüksekgönül, M., & Baydoğan, M. G. (2021). Learning prototypes for multiple instance learning. Turkish Journal of Electrical Engineering & Computer Sciences, 29(7), 2901–2919. https://doi.org/10.3906/elk-2101-103
• Sousa, T., Silva, A., & Neves, A. (2004). Particle Swarm based Data Mining Algorithms for classification tasks. Parallel Computing, 30(5–6), 767–783. https://doi.org/10.1016/j.parco.2003.12.015
• Taherkhani, M., & Safabakhsh, R. (2016). A novel stability-based adaptive inertia weight for particle swarm optimization. Applied Soft Computing, 38, 281–295. https://doi.org/10.1016/j.asoc.2015.10.004
• Tan, T. Y., Zhang, L., & Lim, C. P. (2019). Intelligent skin cancer diagnosis using improved particle swarm optimization and deep learning models. Applied Soft Computing, 84, 105725. https://doi.org/10.1016/j.asoc.2019.105725
• Telikani, A., Tahmassebi, A., Banzhaf, W., & Gandomi, A. H. (2022). Evolutionary Machine Learning: A Survey. ACM Computing Surveys, 54(8), 1–35. https://doi.org/10.1145/3467477
• Yilmaz, A., & Yolcu, U. (2022). Dendritic neuron model neural network trained by modified particle swarm optimization for time‐series forecasting. Journal of Forecasting, 41(4), 793–809. https://doi.org/10.1002/for.2833
• Yin, P.-Y., Glover, F., Laguna, M., & Zhu, J.-X. (2011). A Complementary Cyber Swarm Algorithm. International Journal of Swarm Intelligence Research, 2(2), 22–41. https://doi.org/10.4018/jsir.2011040102
• Yuan, G.-X., Ho, C.-H., & Lin, C.-J. (2012). Recent Advances of Large-Scale Linear Classification. Proceedings of the IEEE, 100(9), 2584–2603. https://doi.org/10.1109/JPROC.2012.2188013