Voting Combinations-Based Ensemble: A Hybrid Approach

Abstract

Machine learning (ML) is a prominent and extensively researched field in the artificial intelligence area which assists to strengthen the accomplishment of classification. In this study, the main idea is to provide the classification and analysis of ML and Ensemble Learning (EL) algorithms. To support this idea, six supervised ML algorithms, C4.5 (J48), K-Nearest Neighbor (KNN), Logistic Regression (LR), Support Vector Machine (SVM), Naive Bayes (NB) and One Rule (OneR) in addition the five UCI Datasets of ML Repository, are being applied that demonstrates the robustness and effectiveness of numerous approaches. In this paper, a voting-based ensemble classifier has been proposed along with two base learners (namely, Random Forest and Rotation Forest) to progress the performance. Whereas, for analytical procedures, significant parameters have been considered: Accuracy, Area under Curve (AUC), recall, precision, and F-measure values. Hence, the prime objective of this research is to obtain binary classification and efficiency by conducting the progress of ML and EL approaches. We present experimental outcomes that validate the effectiveness of our method to well-known competitive approaches. Image recognition and ML challenges, such as binary classification, can be solved using this method.

Keywords

• Artificial Intelligence
• Data Mining
• Classification
• machine learning
• Pattern Recognition

References

1. 1. Accorsi R, Manzini R, Pascarella P, Patella M, Sassi S. “Data Mining and Machine Learning for Condition-based Maintenance”. Procedia manufacturing, 11,1153–1161, 2017. 2. Shao Y, Liu Y, Ye X, Zhang S. “A Machine Learning based global simulation data mining approach for efficient design changes”. Advances in Engineering Software,124, 22–41, 2018.
2. 3. Hüllermeier E. “Fuzzy sets in Machine Learning and data mining”. Applied Soft Computing, 11(2). 1493–1505, 2011.
3. 4. Kavakiotis I, Tsave O, Salifoglou A, Maglaveras N, Vlahavas I, Chouvarda I." Machine Learning and data mining methods in diabetes research". Computational and structural biotechnology journal, 15, 104-116, 2017.
4. 5. Shafiq M, Tian Z, Bashir AK, Jolfaei A, Yu X." Data mining and Machine Learning methods for sustainable smart cities traffic classification: a survey". Sustainable Cities and Society, 60, 102177, 2020
5. 6. Deepajothi S, Selvarajan S. "A comparative study of classification techniques on adult data set". International Journal of Engineering Research & Technology (IJERT), 1, 2012.
6. 7. Bansal D, Chhikara R, Khanna K, Gupta P. "Comparative analysis of various Machine Learning algorithms for detecting dementia". Procedia computer science, 132, 1497-1502, 2018.
7. 8. Wang X, Zhou C, Xu X. "Application of C4. 5 decision tree for scholarship evaluations". Procedia Computer Science, 151, 179-184,2019.
8. 9. Mohammed M, Mwambi H, Mboya, IB, Elbashir MK, & Omolo B. "A stacking ensemble deep learning approach to cancer type classification based on TCGA data. Scientific reports", 11(1), 1-22, 2021.

9. 10. Nusinovici S, Tham YC, Yan MYC , Ting DSW, Li J, Sabanayagam C, Cheng CY. "Logistic regression was as good as Machine Learning for predicting major chronic diseases" Journal of clinical epidemiology, 122, 56-69, 2020.
10. 11. Xu F, Pan Z, Xia R. "E-commerce product review sentiment classification based on a naïve Bayes continuous learning framework". Information Processing & Management, 57(5), 102221,2020.
11. 12. Wang C, Du J, Chen G, Wang H, Sun L, Xu K, He Z. "QAM classification methods by SVM Machine Learning for improved optical interconnection. " Optics Communications, 444, 1-8,2019.
12. 13. Tama BA, & Lim S. "Ensemble learning for intrusion detection systems: A systematic mapping study and cross-benchmark evaluation", Computer Science Review, 39, 100357, 2021.
13. 14. Abro AA, Yimer MA, Bhatti Z. "Identifying the Machine Learning Techniques for Classification of Target Datasets". Sukkur IBA Journal of Computing and Mathematical Sciences, 4(1), 45-52,2020.
14. 15. ABRO AA, TAŞCI E, UGUR A. "A Stacking-based Ensemble Learning Method for Outlier Detection". Balkan Journal of Electrical and Computer Engineering, 8(2), 181-185,2020
15. 16. ABRO AA. "Vote-Based: Ensemble Approach". Sakarya University Journal of Science, 25(3), 871-879, 2021.
16. 17. Mantas CJ, Abellán J, Castellano JG. "Analysis of Credal-C4. 5 for classification in noisy domains. Expert Systems with Applications". 61, 314-326, 2016.
17. 18. Pavlyshenko B. "Using stacking approaches for machine learning models", IEEE Second International Conference on Data Stream Mining & Processing, 255-258, 2018.
18. 19. Sikora R. "A modified stacking ensemble machine learning algorithm using genetic algorithms", In Handbook of research on organizational transformations through big data analytics, 43-53, 2015.
19. 20. Tan Y,Shenoy PP. "A bias-variance based heuristic for constructing a hybrid logistic regression-naïve Bayes model for classification". International Journal of Approximate Reasoning, 117, 15-28, 2020.
20. 21. Chen S, Webb GI, Liu L, Ma X. "A novel selective naïve Bayes algorithm". Knowledge-Based Systems, 192, 105361, 2020.
21. 22. Utkin LV. "An imprecise extension of SVM-based Machine Learning models". Neurocomputing, 331, 18-32, 2019.
22. 23. Singh BK, Verma K, Thoke AS. "Investigations on impact of feature normalization techniques on classifier's performance in breast tumor classification". International Journal of Computer Applications, 116(19), 2017.
23. 24. Kumar AD, Selvam RP, & Palanisamy V. "Hybrid classification algorithms for predicting student performance", International Conference on Artificial Intelligence and Smart Systems, 1074-1079, 2021.
24. 25. Zareapoor M, & Shamsolmoali P. "Application of credit card fraud detection: Based on bagging ensemble classifier", Procedia computer science, 48(2015), 679-685, 2015..
25. 26. Van der Heide EMM, Veerkamp RF, Van Pelt ML, Kamphuis, C, Athanasiadis I, Ducro BJ. "Comparing regression, naive Bayes, and random forest methods in the prediction of individual survival to second lactation in Holstein cattle". Journal of dairy science, 102(10), 9409-9421, 2019.
26. 27. Chen Y. "Mining of instant messaging data in the Internet of Things based on support vector machine" Computer Communications, 154, 278-287., 2020., 2020.
27. 28. Nevill-Manning CG, Holmes G, Witten IH. "The development of Holte's 1R classifier". In Proceedings 1995 Second New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems, 239-242,1995.
28. 29. Dua D, Graff C. “UCI Machine Learning Repository”. http://archive.ics.uci.edu/ml (9.07.2021).
29. 30. Engel TA, Charão AS, Kirsch-Pinheiro M, Steffenel LA. "Performance improvement of data mining in Weka through GPU acceleration". Procedia Computer Science, 32, 93-100,2014.
30. 31. Abro, A. A., Siddique, W. A., Talpur, M. S. H., Jumani, A. K., & Yaşar, E. “A combined approach of base and meta learners for hybrid system”. Turkish Journal of Engineering, 7(1), 25-32, 2023.
31. 32. Abro, A. A., Khan, A. A., Talpur, M. S. H., & Kayijuka, I. “Machine Learning Classifiers: A Brief Primer”. University of Sindh Journal of Information and Communication Technology, 5(2), 63-68, 2021.
32. 33. Chandio, J. A., Talpur, M. S. H., Abro, A. A., Bux, H., Khokhar, N. U. A. A., Shah, A. A., & Saima, M. “Study Of Customers Perception About Shopping Trend Involving E-Commerce: A Comparative Study”. Turkish Online Journal of Qualitative Inquiry, 12(8), 5415-5424, 2021.