A comparative study of ensemble methods in the field of education: Bagging and Boosting algorithms

Abstract

This study aims to conduct a comparative study of Bagging and Boosting algorithms among ensemble methods and to compare the classification performance of TreeNet and Random Forest methods using these algorithms on the data extracted from ABİDE application in education. The main factor in choosing them for analyses is that they are Ensemble methods combining decision trees via Bagging and Boosting algorithms and creating a single outcome by combining the outputs obtained from each of them. The data set consists of mathematics scores of ABİDE (Academic Skills Monitoring and Evaluation) 2016 implementation and various demographic variables regarding students. The study group involves 5000 students randomly recruited. On the deletion of loss data and assignment procedures, this number decreased to 4568. The analyses showed that the TreeNet method performed more successfully in terms of classification accuracy, sensitivity, F1-score and AUC value based on sample size, and the Random Forest method on specificity and accuracy. It can be alleged that the TreeNet method is more successful in all numerical estimation error rates for each sample size by producing lower values compared to the Random Forest method. When comparing both analysis methods based on ABİDE data, considering all the conditions, including sample size, cross validity and performance criteria following the analyses, TreeNet can be said to exhibit higher classification performance than Random Forest. Unlike a single classifier or predictive method, the classification or prediction of multiple methods by using Boosting and Bagging algorithms is considered important for the results obtained in education.

Keywords

• Educational Data Mining
• Ensemble Learning
• Bagging
• Boosting

References

1. Abdar, M., Zomorodi-Moghadam, M., & Zhou, X. (2018, 12-14, November). An ensemble-based decision tree approach for educational data mining [Conference presentation]. In 2018 5th International Conference on Behavioral, Economic, and Socio-Cultural Computing (BESC), Kaohsiung, Taiwan. https://doi.org/10.1109/BESC.2018.8697318
2. Abeel, T., Helleputte, T., Van de Peer, Y., Dupont, P., & Saeys, Y. (2010). Robust biomarker identification for cancer diagnosis with ensemble feature selection methods. Bioinformatics, 26(3). 392-398. https://doi.org/10.1093/bioinformatics/btp630
3. Abidi, S.M.R., Zhang, W., Haidery, S.A., Rizvi, S.S., Riaz, R., Ding, H., & Kwon, S.J. (2020). Educational sustainability through big data assimilation to quantify academic procrastination using ensemble classifiers. Sustainability, 12(15), 6074. https://doi.org/10.3390/su12156074
4. Aggarwal, D., Mittal, S., & Bali, V. (2021). Significance of non-academic parameters for predicting student performance using ensemble learning techniques. International Journal of System Dynamics Applications, 10(3), 38 49. https://doi.org/10.4018/IJSDA.2021070103
5. Akman, M. (2010). An overview of data mining techniques and analysis of Random Forests method: An application on medical field [Unpublished master’s thesis]. Ankara University.
6. Almasri, A., Celebi, E., & Alkhawaldeh, R.S. (2019). EMT: Ensemble meta-based tree model for predicting student performance. Hindawi, 1 13. https://doi.org/10.1155/2019/3610248
7. Amrieh, E.A., Hamtini, T., & Aljarah, I. (2016). Mining educational data to predict student’s academic performance using ensemble methods. International Journal of Database Theory and Application, 9(8), 119-136. http://dx.doi.org/10.14257/ijdta.2016.9.8.13
8. Ashraf, M., Zaman, M., & Ahmed, M. (2020). An intelligent prediction system for educational data mining based on ensemble and filtering approaches. Procedia Computer Science, 167, 1471-1483. https://doi.org/10.1016/j.procs.2020.03.358

9. Ashraf, M., Salal, Y.K., & Abdullaev, S.M. (2021). Educational Data Mining Using Base (Individual) and Ensemble Learning Approaches to Predict the Performance of Students. In Data Science. Springer. https://doi.org/10.1007/978-981-16-1681-5_2
10. Arun, D.K., Namratha, V., Ramyashree, B.V., Jain, Y.P., & Choudhury, A.R. (2021, 27-29, January). Student academic performance prediction using educational data mining [Conference presentation]. In 2021 International Conference on Computer Communication and Informatics (ICCCI), Coimbatore, India. https://doi.org/10.1109/ICCCI50826.2021.9457021
11. Baskin, I.I., Marcou, G., Horvath, D., & Varnek, A. (2017a). Bagging and boosting of classification models. Tutorials in Chemoinformatics, 241 247. John Wiley & Sons Ltd. https://doi.org/10.1002/9781119161110.ch15
12. Baskin, I.I., Marcou, G., Horvath, D., & Varnek, A. (2017b). Bagging and boosting of regression models. Tutorials in Chemoinformatics, 249-255. John Wiley & Sons Ltd. https://doi.org/10.1002/9781119161110.ch16
13. Bauer, E., & Kohavi, R. (1999). An empirical comparison of voting classification algorithms: Bagging. Boosting and variants. Machine Learning. 36(1), 105 139. https://doi.org/10.1023/A:1007515423169
14. Biau, G. (2012). Analysis of a Random Forest. Journal of Machine Learning Research, 13(2012), 1063-1095. https://www.jmlr.org/papers/volume13/biau12a/biau12a.pdf
15. Biau, G., & Scornet, E., (2016). A random forest guided tour. An Official Journal of the Spanish Society of Statistics and Operations Research, 25(2), 197 227. https://doi.org/10.1007/s11749-016-0481-7
16. Breiman, L. (1996). Bagging predictors. Machine Learning 24(2), 123 140. https://doi.org/10.1007/BF00058655
17. Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5 32. https://doi.org/10.1023/A:1010933404324
18. Chen, T., & Guestrin, C. (2016, 13, August). Xgboost: A scalable tree boosting system [Conference presentation]. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, San Francisco, CA, USA. http://dx.doi.org/10.1145/2939672.2939785
19. Clarke, B., Fokoue, E., & Zhang, H.H. (2009). Principles and theory for data mining and machine learning. Springer Science & Business Media. https://doi.org/10.1007/978-0-387-98135-2
20. Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk, Ş. (2012). Multivariate statistics for social sciences: SPSS and LISREL applications (2th edition). Pegem Academy.
21. Do-Nascimento, R.L., Fagundes, R.A., & Maciel, A.M. (2019, 15-18, July). Prediction of School Efficiency Rates through Ensemble Regression Application [Conference presentation]. In 2019 IEEE 19th International Conference on Advanced Learning Technologies, Maceio, Brazil. https://doi.org/10.1109/ICALT.2019.00050
22. Dietterich, T.G. (2000a). Ensemble Methods in Machine Learning. In: Multiple Classifier Systems. MCS 2000. Lecture Notes in Computer Science, 1857, 1-15. https://doi.org/10.1007/3-540-45014-9_1
23. Dietterich, T.G. (2000b). An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization. Machine Learning, 40(2), 139-157. https://doi.org/10.1023/A:1007607513941
24. Dietterich, T.G. (2002). Ensemble learning. The Handbook of Brain Theory and Neural Networks, 2(1), 110-125. https://courses.cs.washington.edu/courses/cse446/12wi/tgd-ensembles.pdf
25. Efron, B., & Tibshirani, R. (1993). An Introduction to the Bootstrap. Chapman and Hall/CRC.
26. Elish, M.O., & Elish, K.O. (2009, 24-27, March). Application of treenet in predicting object-oriented software maintainability: A comparative study. In 2009 13th European Conference on Software Maintenance and Reengineering, Kaiserslautern, Germany. https://doi.org/10.1109/CSMR.2009.57
27. Ferreira, A.J., & Figueiredo, M.A. (2012). Boosting algorithms: A review of methods, theory, and applications. Ensemble machine learning (1th edition, 35-85). Springer. https://doi.org/10.1007/978-1-4419-9326-7_2
28. Freund, Y., & Schapire, R.E. (1996, 3-6, July). Experiments with a new boosting algorithm [Conference presentation]. Proceedings of the Thirteenth International Conference on International Conference on Machine Learning, Bari Italy.
29. Friedman, J., Hastie, T., & Tibshirani, R. (2000). Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors). Annals of Statistics, 28(2), 337-407. https://doi.org/10.1214/aos/1016218223
30. Friedman, J.H. (2001). Greedy function approximation: a gradient boosting machine. Annals of Statistics, 29(5) 1189-1232. https://www.jstor.org/stable/2699986
31. Friedman, J.H. (2002). Stochastic gradient boosting. Computational Statistics & Data Analysis, 38(4), 367-378. https://doi.org/10.1016/S0167-9473(01)00065-2
32. Friedman, J.H., & Meulman, J. J. (2003). Multiple additive regression trees with application in epidemiology. Statistics in Medicine, 22(9), 1365-1381. https://doi.org/10.1002/sim.1501
33. Geneur, R., Poggi, J.M., Tuleao Malot, C., & Villa-Vialaneix, N. (2017). Random forest for big data. Big Data Research, 9, 28-46. https://doi.org/10.1016/j.bdr.2017.07.003
34. Geron, A. (2019). Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems (1th edition). O'Reilly Media.
35. Guo, J., Bai, L., Yu, Z., Zhao, Z., & Wan, B. (2021). An AI-application-oriented in-class teaching evaluation model by using statistical modeling and ensemble learning. Sensors, 21(1), 241. https://doi.org/10.3390/s21010241
36. Han, J., Kamber, M., & Pei, J., (2012). Data mining: concepts and techniques (3th edition). Elsevier.
37. Hansen, L.K., & Salamon, P. (1990). Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10), 993-1001. https://doi.org/10.1109/34.58871
38. Hastie, T., Tibshirani, R. & Friedman, J.H. (2009). The elements of statistical learning: data mining, inference, and prediction. Springer. https://doi.org/10.1007/978-0-387-21606-5
39. Hill, T., & Lewicki, P. (2006). Statistics: methods and applications: a comprehensive reference for science, industry, and data mining (1th edition). StatSoft, Inc.
40. Ho, T.K. (1998). The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(8), 832-844. https://doi.org/10.1109/34.709601
41. Huffer, F.W., & Park, C. (2020). A Simple Rule for Monitoring the Error Rate of Random Forest for Classification. Quantitative Bio-Science, 39(1), 1-15.
42. Injadat, M., Moubayed, A., Nassif, A.B., & Shami, A. (2020a). Systematic ensemble model selection approach for educational data mining. Knowledge-Based Systems, 200, 105992. https://doi.org/10.1016/j.knosys.2020.105992
43. Injadat, M., Moubayed, A., Nassif, A.B., & Shami, A. (2020b). Multi-split optimized bagging ensemble model selection for multi-class educational data mining. Applied Intelligence, 50(12), 4506-4528. https://doi.org/10.1007/s10489-020-01776-3
44. Kapucu, C., & Cubukcu, M. (2021). A supervised ensemble learning method for fault diagnosis in photovoltaic strings. Energy, 227, 1-12. https://doi.org/10.1016/j.energy.2021.120463
45. Karalar, H., Kapucu, C., & Gürüler, H. (2021). Predicting students at risk of academic failure using ensemble model during pandemic in a distance learning system. International Journal of Educational Technology in Higher Education, 18(1), 1-18. https://doi.org/10.1186/s41239-021-00300-y
46. Kausar, S., Oyelere, S., Salal, Y., Hussain, S., Cifci, M., Hilcenko, S., ... & Huahu, X. (2020). Mining smart learning analytics data using ensemble classifiers. International Journal of Emerging Technologies in Learning, 15(12), 81 102. https://www.learntechlib.org/p/217561/
47. Keser, S.B., & Aghalarova, S. (2022). HELA: A novel hybrid ensemble learning algorithm for predicting academic performance of students. Education and Information Technologies, 27(4), 4521-4552. https://doi.org/10.1007/s10639-021-10780-0
48. Kotsiantis, S., Patriarcheas, K., & Xenos, M. (2010). A combinational incremental ensemble of classifiers as a technique for predicting students’ performance in distance education. Knowledge Based Systems, 23(6), 529 535. https://doi.org/10.1016/j.knosys.2010.03.010
49. Kumari, G.T. (2012). A Study of Bagging and Boosting approaches to develop meta-classifier. Engineering Science and Technology: An International Journal, 2(5), 850-855.
50. Leedy, P.D., & Ormrod, J.E. (2005). Practical research (Vol. 108). Saddle River.
51. Lee, S.L.A., Kouzani, A.Z., & Hu, E. J. (2010). Random forest based lung nodule classification aided by clustering. Computerized Medical Imaging and Graphics, 34(7), 535-542. https://doi.org/10.1016/j.compmedimag.2010.03.006
52. Li, B., Yu, Q., & Peng, L. (2022). Ensemble of fast learning stochastic gradient boosting. Communications in Statistics-Simulation and Computation, 51(1), 40-52. https://doi.org/10.1080/03610918.2019.1645170
53. Machová, K., Puszta, M., Barčák, F., & Bednár, P. (2006). A comparison of the bagging and the boosting methods using the decision trees classifiers. Computer Science and Information Systems, 3(2), 57-72. https://doi.org/10.2298/CSIS0602057M
54. Maclin, R., & Opitz, D. (1997, 27-31, July). An empirical evaluation of bagging and boosting [Conference presentation]. AAAI-97: Fourteenth National Conference on Artificial Intelligence, Rhode Island.
55. Märker, M., Pelacani, S., & Schröder, B. (2011). A functional entity approach to predict soil erosion processes in a small Plio-Pleistocene Mediterranean catchment in Northern Chianti, Italy. Geomorphology, 125(4), 530 540. https://doi.org/10.1016/j.geomorph.2010.10.022
56. Mi, C., Huettmann, F., Guo, Y., Han, X., & Wen, L. (2017). Why choose Random Forest to predict rare species distribution with few samples in large undersampled areas? Three Asian crane species models provide supporting evidence. Peer J, 5, e2849.
57. Mousavi, R., & Eftekhari, M. (2015). A new ensemble learning methodology based on hybridization of classifier ensemble selection approaches. Applied Soft Computing, 37, 652-666. https://doi.org/10.1016/j.asoc.2015.09.009
58. Nisbet, R., Elder, J., & Miner, G. (2009). Handbook of statistical analysis and data mining applications (1th edition). Academic Press.
59. Olson, D.L., & Delen, D. (2008). Advanced data mining techniques. Springer Science & Business Media.
60. Onan, A. (2015). On the performance of ensemble learning for automated diagnosis of breast cancer. R. Silhavy R. Senkerik, Z. K. Oplatkova, Z. Prokopova, & P. Silhavy (eds.), In Artificial Intelligence Perspectives and Applications: Proceedings of the 4th Computer Science On-line Conference, Vol 1 (pp. 119-129). Springer International Publishing.. https://doi.org/10.1007/978-3-319-18476-0_13
61. Opitz, D.W., & Shavlik, J.W. (1996). Generating accurate and diverse members of a neural network ensemble. Advances in Neural Information Processing Systems, 8, 535-541.
62. Padmaja, B., Prasad, V.R., & Sunitha, K.V.N. (2016). TreeNet analysis of human stress behavior using socio mobile data. Journal of Big Data, 3(1), 1 15. https://doi.org/10.1186/s40537-016-0054-3
63. Padmaja, B., Srinidhi, C., Sindhu, K., Vanaja, K., Deepika, N.M., & Patro, E.K.R. (2021). Early and accurate prediction of heart disease using machine learning model. Turkish Journal of Computer and Mathematics Education, 12(6), 4516-4528.
64. Polikar, R. (2006). Ensemble based systems in decision making. IEEE Circuits and Systems Magazine, 6(3). 21-45. https://doi.org/10.1109/MCAS.2006.1688199
65. Polikar, R. (2012). Ensemble learning. In Ensemble machine learning (1th edition pp. 1-34). Springer. https://doi.org/10.1007/978-1-4419-9326-7_1
66. Premalatha, N., & Sujatha, S. (2021, 15-17, September). An Effective Ensemble Model to Predict Employment Status of Graduates in Higher Educational Institutions [Conference presentation]. In 2021 Fourth International Conference on Electrical, Computer and Communication Technologies Erode, India. https://doi.org/10.1109/icecct52121.2021.9616952
67. Probst, P., & Boulesteix, A.L. (2017). To tune or not to tune the number of trees in random forest. The Journal of Machine Learning Research, 18(1), 6673-6690. http://jmlr.org/papers/v18/17-269.html
68. Rokach, L. (2019). Ensemble learning: Pattern classification using ensemble methods (2th edition). World Scientific. https://doi.org/10.1142/9789811201967_0003
69. Pong-Inwong, C., & Kaewmak, K. (2016, 14-17, October). Improved sentiment analysis for teaching evaluation using feature selection and voting ensemble learning integration [Conference presentation]. In 2016 2nd IEEE international conference on computer and communications, Chengdu, China. https://doi.org/10.1109/CompComm.2016.7924899
70. Quinlan, J.R. (1996, 4-8, August). Bagging, boosting, and C4. 5 [Conference presentation]. In 13th National Conference on Artificial Intelligence, Portland, Oregon, USA.
71. Saeys, Y., Abeel, T., & Peer, Y.V.D. (2008). Robust feature selection using ensemble feature selection techniques. W. Daelemans, B. Goethals & K. Morik (Eds.), Machine learning and knowledge discovery in databases (pp 313 325) Springer. https://doi.org/10.1007/978-3-540-87481-2_21
72. Sagi, O., & Rokach, L. (2018). Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery. 8(4). e1249. https://doi.org/10.1002/widm.1249
73. Schapire, R.E. (2003). The boosting approach to machine learning: An overview. Nonlinear Estimation and Classification, 149-171. https://doi.org/10.1007/978-0-387-21579-2_9
74. Schroeder, M.A., Lander, J., & Levine-Silverman, S. (1990). Diagnosing and dealing with multicollinearity. Western Journal of Nursing Research, 12(2), 175-187. https://doi.org/10.1177/019394599001200204
75. Sinharay, S. (2016). An NCME instructional module on data mining methods for classification and regression. Educational Measurement: Issues and Practice, 35(3), 38-54. https://doi.org/10.1111/emip.12115
76. Skurichina, M., & Duin, R.P. (2002). Bagging, boosting and the random subspace method for linear classifiers. Pattern Analysis & Applications, 5(2), 121 135. https://doi.org/10.1007/s100440200011
77. Steinki, O., & Mohammad, Z. (2015). Introduction to ensemble learning. Available at SSRN, 1(1), 1-9. http://dx.doi.org/10.2139/ssrn.2634092
78. Strobl, C., Malley, J., & Tutz, G. (2009). An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological Methods, 14(4), 323. https://doi.org/10.1037/a0016973
79. Subasi, A., El-Amin, M.F., Darwich, T., & Dossary, M. (2022). Permeability prediction of petroleum reservoirs using stochastic gradient boosting regression. Journal of Ambient Intelligence and Humanized Computing, 13, 3555-3564. https://doi.org/10.1007/s12652-020-01986-0
80. Sutton, C.D. (2005). Classification and regression trees, bagging, and boosting. Handbook of Statistics, 24, 303-329. https://doi.org/10.1016/S0169-7161(04)24011-1
81. Şevgin, H. (2020). Predicting the ABIDE 2016 science achievement: The comparison of MARS and BRT data mining methods [Unpublished Doctoral Thesis]. Gazi University.
82. Şevgin, H., & Önen, E. (2022). Comparison of Classification Performances of MARS and BRT Data Mining Methods: ABİDE-2016 Case. Education and Science, 47(211). http://dx.doi.org/10.15390/EB.2022.10575
83. Tabachnick, B.G., & Fidell, L.S. (2015). Using multivariate statistics (6th edition). (M. Baloğlu, Trans.). Nobel Publications. (Original work published 2012).
84. Ting, K. M. (2017). Confusion matrix. In C. Sammut & G. I. Webb (Eds.) Encyclopedia of Machine Learning and Data Mining (pp. 260–260). Springer.
85. Tuğ Karoğlu, T.T., & Okut, H., (2020). Classification of the placement success in the undergraduate placement examination according to decision trees with bagging and boosting methods. Cumhuriyet Science Journal, 41(1), 93 105. https://doi.org/10.17776/csj.544639
86. Wang, Z., Wang, Y., & Srinivasan, R.S. (2018). A novel ensemble learning approach to support building energy use prediction. Energy and Buildings, 159, 109 122. https://doi.org/10.1016/j.enbuild.2017.10.085
87. Yurdugül, H. (2006). The comparison of reliability coefficients in parallel, tau-equivalent, and congeneric measurements. Ankara University Journal of Faculty of Educational Sciences, 39(1), 15-37. https://doi.org/10.1501/Egifak_0000000127
88. Zhang, C., & Ma, Y. (2012). Ensemble machine learning: methods and applications. Springer. https://doi.org/10.1007/978-1-4419-9326-7
89. Zhou Z.H. (2012). Ensemble methods: foundations and algorithms. Chapman and Hall/CRC.