Comparison of the Performance of Gradient Boosting and Extreme Gradient Boosting Methods in Classifying Timms Science Achievement

Abstract

This study aims to compare the classification performance of machine learning methods Gradient Boosting (GB) and Extreme Gradient Boosting (XGBoost). The Trends in International Mathematics and Science Study 2019 (TIMSS 2019) science data set was used in the study. The dataset consists of data collected from a total of 2565 students, 1309 of whom are girls (51%) and 1256 (49%) are boys. A Python-based program was used for data analysis. In the study, Area Under the Curve (AUC), accuracy, precision, recall, F1 score, Matthews correlation coefficient (MCC), and training time were used as performance indicators. The study revealed that hyperparameter tuning had a positive impact on the performance of both methods. The analysis results show that the GB method was more successful compared to the XGBoost method in all performance measures except for training time. According to the GB method, 'student confidence in science' was identified as the most influential factor in science achievement, while the XGBoost method highlighted 'home educational resources' as the most significant predictor.

Keywords

• Timms
• Science
• Gradient Boosting
• Extreme Gradient Boosting
• Machine Learning

Supporting Institution

No support was received from any individuals, institutions, or organizations in the conduct of this study.

Ethical Statement

The current study is not a study requiring ethics committee approval since it was prepared using an open access dataset.

References

1. Sarker, I. H. “Machine learning: Algorithms, real-world applications and research directions”, SN computer science, vol.2(3), 160, pp. 1-21,2021.
2. Nasteski, V.” An overview of the supervised machine learning methods”. Horizons. b, 4(51-62), 56, 2017.
3. Baştanlar, Y. & Özuysal, M. “Introduction to machine learning”. miRNomics: MicroRNA biology and computational analysis, 105-128, 2014.
4. Chang, Y.C. & Bangsri, A. “Thai Students’ Perceived Teacher Support on Their Reading Ability: Mediating Effects of Self-Efficacy and Sense of School Belonging”. International Journal of Educational Methodology. 6(2), 435 – 446, 2020.
5. Juan, A., Hannan, S. & Namome, C. “I believe I can do science: Self-efficacy and science achievement of Grade 9 students in South Africa”. South African Journal of Science, 114(7-8), 48-54, 2018.
6. Eser, M. T. & Çobanoğlu Aktan, D. “Educational data mining: The analysis of the factors affecting science instruction by clustering analysis”. International Journal of Educational Methodology, 7(3), 487-500, 2021
7. Torney-Purta, J. & Amadeo, J. A. “International large-scale assessments: Challenges in reporting and potentials for secondary analysis”. Research in Comparative and International Education, 8(3), 248- 258, 2013.
8. Turkey Ministry of National Education (TMNE). ” TIMSS 2015 national math and science preliminary report 4th and 8th grades”. Ankara: MEB: Measurement. General Directorate of Evaluation and Examination Services, 2020.

9. Aydogan, I. & Gelbal, S. “Determination of the characteristics predicting science achievement through the classification and regression tree (cart) method: The case of TIMSS 2015 Turkey”. Egitim ve Bilim-Education and Science, 47(209), 2022.
10. Hooper, M., Mullis, I. V. S. & Martin, M. O. “TIMSS 2015 contex questionnaire framework”. I. V. S. Mullis and M. O. Martin (Ed.), TIMSS 2015 assessment frameworks içinde (s. 61-82). Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College, 2013.
11. Akyüz, G. “The effects of student and school factors on mathematics achievement in TIMSS 2011”, Education and Science, Vol. 39, No 172, 2014.
12. Akyuz, G. & Berberoglu, G. “Teacher and Classroom Characteristics and Their Relations to Mathematics Achievement of the Students in the TIMSS”. New Horizons in Education, 58(1), 77-95, 2010.
13. Yalcin, S., Demirtasli, R.N., Dibek, M. I. & , Yavuz, H. C. “The Effect of Teacher and Student Characteristics on TIMSS 2011 Mathematics Achievement of Fourth-and Eighth-Grade Students in Turkey”. International Journal of Progressive Education, 13(3), 79-94, 2017.
14. Kilic, S. & Askin, Ö. E. “Parental influence on students’ mathematics achievement: The comparative study of Turkey and best performer countries in TIMSS 2011”. Procedia-social and behavioral sciences, 106, 2000-2007, 2013.
15. Sandoval-Hernández, A. & Białowolski, P. “Factors and conditions promoting academic resilience: a TIMSS-based analysis of five Asian education systems”. Asia Pacific Education Review, 17(3), 511-520, 2016.
16. Topçu, M. S., Erbilgin, E. & Arıkan, S. “Factors predicting Turkish and Korean students' science and mathematics achievement in TIMSS 2011”. Eurasia Journal of Mathematics, Science & Technology Education, 12(7), 1711-1737 2016.
17. Nilsen, T., Kaarstein, H. & Lehre, A. C. “Trend analyses of TIMSS 2015 and 2019: school factors related to declining performance in mathematics”. Large-scale Assessments in Education, 10(1), 1-19, 2022.
18. Askin, O. E. & Gokalp, F. “Comparing the predictive and classification performances of logistic regression and neural networks: a case study on Timss 2011”. Procedia-Social and Behavioral Sciences, 106, 667-676, 2013.
19. Depren, S. K., Aşkın, Ö. E. & Öz, E. “Identifying the classification performances of educational data mining methods: A case study for TIMSS”. Educational Sciences: Theory & Practice, 17(5), 2017.
20. Siemssen, A. M. “Using Data Mining to Model Student Achievement on the 4 th Grade TIMSS 2015 Mathematics Assessment: A Five Nation Study”. The University of Texas at El Paso, 2018.
21. Filiz, E. & Öz, E. “Educational data mining methods for TIMSS 2015 mathematics success: Turkey case”. Sigma Journal of Engineering and Natural Sciences, 38(2), 963-977, 2020.
22. Filiz, E. & Öz, E. “Finding the Best Algorithms and Effective Factors in Classifıcation of Turkish Science Student Success”. Journal of Baltic Science Education 18, 239-253, 2019.
23. Topal, K. H. “Variable Selection via the Adaptive Elastic Net: Mathematics Success of the Students in Singapore and Turkey”. Journal of Applied Microeconometrics (JAME). 1(1), 40-54, 2021,
24. Bezek Güre, Ö. “Investigation of ensemble methods in terms of statistics: TIMMS 2019 example. Neural Computing and Applications, 35(32), 23507-23520, 2023.
25. Şevgin, H. & Eranıl, A. K. “Investigation of Turkish Students' School Engagement through Random Forest Methods Applied to TIMSS 2019: A Problem of School Psychology”. International Journal of Psychology and Educational Studies, 10(4), 896-909, 2023.
26. Wang, F., King, R. B. & Leung, S. O. “Why do East Asian students do so well in mathematics? A machine learning study”. International Journal of Science and Mathematics Education, 21(3), 691-711, 2023.
27. Anıl, D.” Uluslararası öğrenci başarılarını değerlendirme programı (PISA)’nda Türkiye’deki öğrencilerin fen bilimleri başarılarını etkileyen faktörler”. Eğitim ve Bilim, 34(152), 2010.
28. Bernardo, A. B., Cordel, M. O., Lapinid, M. R. C., Teves, J. M. M., Yap, S. A. &, Chua, U. C. “Contrasting profiles of low-performing mathematics students in public and private schools in the Philippines: insights from machine learning”. Journal of Intelligence, 10(3), 61, 2022.
29. Güre, Ö. B., Kayri, M. & Erdoğan, F. “Analysis of Factors Effecting PISA 2015 Mathematics Literacy via Educational Data Mining”. Education & Science/Egitim ve Bilim, 45(202), 2020.
30. Immekus, J. C., Jeong, T. S. & Yoo, J. E. “Machine learning procedures for predictor variable selection for schoolwork-related anxiety: evidence from PISA 2015 mathematics, reading, and science assessments”. Large-scale Assessments in Education, 10(1), 30, 2022.
31. Karaboğa, H. A., Akogul, S. & Demir, I. “Classification of Students’ Mathematical Literacy Score Using Educational Data Mining: PISA 2015 Turkey Application”. Cumhuriyet Science Journal, 43(3), 543-549, 2022.
32. Lezhnina, O. & Kismihók, G. “Combining statistical and machine learning methods to explore German students’ attitudes towards ICT in PISA”. International Journal of Research & Method in Education, 45(2), 180-199, 2022.
33. Şevgin, H. & Önen, E. “MARS ve BRT Veri Madenciliği Yöntemlerinin Sınıflama Performanslarının Karşılaştırılması: ABİDE-2016 Örneği”. Eğitim ve Bilim, 47(211), 2022.
34. Şevgin, H., Güre, Ö. B. & Kayri, M. “An Analysis of MARS and Logistic Regression Methods in Educational Data Mining in Light of Some Performance Indicators”. International Journal of Research in Teacher Education (IJRTE), 14(3), 2023.
35. Berberoglu, G., Celebi, O., Ozdemir, E., Uysal, E. & Yayan, B. “Factors Effecting Achievement Level of Turkish Students in The Third International Mathematics And Science Study (TIMSS)”. Journal of Educational Sciences & Practices, 2(3), 2003.
36. Ceylan, E. & Berberoğlu, G. “Öğrencilerin fen başarısını açıklayan etmenler: Bir modelleme çalışması”. Eğitim ve Bilim, 32(144), 36-48, 2010.
37. Pektaş, M. “Uluslararası matematik ve fen bilimleri eğilimleri çalışması (TIMSS 2007) Türkiye örnekleminde fen bilimleri başarısını etkileyen bazı değişkenlerin incelenmesi “(Master's thesis, Sosyal Bilimler Enstitüsü), 2010.
38. Abazaoğlu, İ. & Taşar, M. F. “Fen bilgisi öğretmen özelliklerinin öğrenci fen başarısı ile ilişkisi: TIMSS 2011 verilerine göre bir durum analizi (Singapur, Güney Kore, Japonya, İngiltere, Türkiye)”. İlköğretim online, 15(3), 2016.
39. Kahraman, N. “Öğrenci katılımı ve TIMSS 2011 fen başarısı arasındaki ilişkinin sınıflara göre karşılaştırılması”. Eğitim ve Bilim, 39(172), 2014.
40. Ölçüoğlu, R. & Çetin, S. “TIMSS 2011 sekizinci sınıf öğrencilerinin matematik başarısını etkileyen değişkenlerin bölgelere göre incelenmesi”. Journal of Measurement and Evaluation in Education and Psychology, 7(1), 202-220, 2016.
41. Pai, P. F., Chen, C. T., Hung, Y. M., Hung, W. Z. & Chang, Y. C. “A group decision classifier with particle swarm optimization and decision tree for analyzing achievements in mathematics and science”. Neural Computing and Applications, 25(7-8), 2011-2023, 2014.
42. Susanto, H., & Utami, E. Comparative analysis to predict reading literacy based on Pisa 2022 using gradient boosted decision trees and extreme gradient boosting. G-Tech: Jurnal Teknologi Terapan, 9(1), 390-399, 2025.
43. Asselman, A., Khaldi, M., & Aammou, S. Enhancing the prediction of student performance based on the machine learning XGBoost algorithm. Interactive Learning Environments, 31(6), 3360-3379, 2023.
44. Saidani, O., Menzli, L. J., Ksibi, A., Alturki, N. & Alluhaidan, A. S. “Predicting student employability through the internship context using gradient boosting models”. IEEE Access, 10, 46472-46489, 2022.
45. Wahyuningsih, T., Iriani, A., Purnomo, H.D. & Sembiring, I. “Predicting students' success level in an examination using advanced linear regression and extreme gradient boosting”. Computer Science and Information Technologies, 5(1), 23-31, 2024.
46. Liu, H., Chen, X. & Liu, X.” Factors influencing secondary school students’ reading literacy: An analysis based on XGBoost and SHAP methods”. Frontiers in Psychology, 13, 948612, 2022.
47. Jeganathan, S., Lakshminarayanan, A. R., Ramachandran, N., & Tunze, G. B. Predicting Academic Performance of Immigrant Students Using XGBoost Regressor. International Journal of Information Technology and Web Engineering (IJITWE), 17(1), 1-19. https://doi.org/10.4018/IJITWE.304052, 2022.
48. Cao, C., Zhang, T. & Xin, T. “The effect of reading engagement on scientific literacy–an analysis based on the XGBoost method”. Frontiers in Psychology, 15, 1329724, 2024.
49. Woo, H. & Kim, J.M. “Impacts of learning orientation on the modeling of programming using feature selection and XGBOOST: A gender-focused analysis”. Applied Sciences, 12(10), 4922, 2022.
50. Yan, K. “Student performance prediction using XGBoost method from a macro perspective”. In 2021 2nd International Conference on Computing and Data Science (CDS) (pp. 453-459). IEEE, 2021.
51. Zopluoglu, C. “Detecting examinees with item preknowledge in large-scale testing using extreme gradient boosting (XGBoost)”. Educational and psychological measurement, 79(5), 931-961, 2019.
52. Çakıt, E. & Dağdeviren, M. “Predicting the percentage of student placement: A comparative study of machine learning algorithms”. Education and Information Technologies, 27(1), 997-1022, 2022.
53. Nirmala, I., Wijayanto, H. & Notodiputro, K.A. “Prediction of Undergraduate Student’s Study Completion Status Using MissForest Imputation in Random Forest and XGBoost Models”. ComTech: Computer, Mathematics and Engineering Applications, 13(1), 53-62, 2022.
54. Guang-yu, L. & Geng, H. “The behavior analysis and achievement prediction research of college students based on XGBoost gradient lifting decision tree algorithm”. In Proceedings of the 2019 7th International Conference on Information and Education Technology (pp. 289-294), 2019.
55. Ridwan, A., Priyatno, A. M., & Ningsih, L. (2024). Predict Students' Dropout and Academic Success with XGBoost. Journal of Education and Computer Applications, 1(2), 1-8.
56. Şevgin, H. & Uçar, R. “An Investigation into the Variables Affecting School Principals’ Burnout through XGBoost Method”. Ahmet Keleşoğlu Eğitim Fakültesi Dergisi, 5(2), 420-439, 2023.
57. Bentéjac, C., Csörgő, A. & Martínez-Muñoz, G. “A comparative analysis of gradient boosting algorithms”. Artificial Intelligence Review, 54, 1937-1967, 2021.
58. Sahin, E. K. “Assessing the predictive capability of ensemble tree methods for landslide susceptibility mapping using XGBoost, gradient boosting machine, and random forest”. SN Applied Sciences, 2(7), 1308, 2020.
59. Sahin, E. K. “Comparative analysis of gradient boosting algorithms for landslide susceptibility mapping”. Geocarto International, 37(9), 2441-2465, 2022.
60. Demir, S. & Sahin, E. K. “An investigation of feature selection methods for soil liquefaction prediction based on tree-based ensemble algorithms using AdaBoost, gradient boosting, and XGBoost”. Neural Computing and Applications, 35(4), 3173-3190, 2023.
61. Sibindi, R., Mwangi, R. W. & Waititu, A. G. “A boosting ensemble learning based hybrid light gradient boosting machine and extreme gradient boosting model for predicting house prices”. Engineering Reports, 5(4), e12599, 2023.
62. Yangın, G. “XGboost ve Karar Ağacı tabanlı algoritmaların diyabet veri setleri üzerine uygulaması” (Master's thesis, Mimar Sinan Güzel Sanatlar Üniversitesi, Fen Bilimleri Enstitüsü), 2019.
63. Natekin, A. & Knoll, A. “Gradient boosting machines, a tutorial”. Frontiers in neurorobotics, 7, 21, 2013.
64. Biau, G., Cadre, B. & Rouvìère, L. “Accelerated gradient boosting”. Machine learning, 108, 971-992, 2019.
65. Zhang, Y. & Haghani, A. “A gradient boosting method to improve travel time prediction”. Transportation Research Part C: Emerging Technologies, 58, 308-324, 2015.
66. Islam, S. & Amin, S. H. “Prediction of probable backorder scenarios in the supply chain using Distributed Random Forest and Gradient Boosting Machine learning techniques”. Journal of Big Data, 7(1), 65, 2020.
67. Son, J. & Yang, S. “A new approach to machine learning model development for prediction of concrete fatigue life under uniaxial compression”. Applied Sciences, 12(19), 9766, 2022.
68. Friedman, J. H. “Greedy function approximation: a gradient boosting machine”. Annals of statistics, 1189-1232, 2001.
69. Osman, A. I. A., Ahmed, A. N., Chow, M. F., Huang, Y. F. & El-Shafie, A. “Extreme gradient boosting (Xgboost) model to predict the groundwater levels in Selangor Malaysia”. Ain Shams Engineering Journal, 12(2), 1545-1556, 2021.
70. Chen, T. & Guestrin, C. “Xgboost: A scalable tree boosting system”. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining (pp. 785-794), 2016.
71. Carmona, P., Climent, F. &, Momparler, A. “Predicting failure in the US banking sector: An extreme gradient boosting approach”. International Review of Economics & Finance, 61, 304-323, 2019.
72. Zhao, Z., Yang, W., Zhai, Y., Liang, Y. & Zhao, Y. “Identify DNA-binding proteins through the extreme gradient boosting algorithm”. Frontiers in Genetics, 12, 821996, 2022.
73. Sikander, R., Ghulam, A. & Ali, F. “XGB-DrugPred: computational prediction of druggable proteins using eXtreme gradient boosting and optimized features set”. Scientific reports, 12(1), 5505, 2022.
74. Keller, P. S., El-Sheikh, M., Granger, D.A. & Buckhalt, J. A. “Interactions between salivary cortisol and alphaamylase as predictors of children’s cognitive functioning and academic performance”. Physiology & Behavior, 105, 987-995, 2012.
75. Akgül, G., Çelik, A.A., Aydın, Z.E. & Öztürk, Z.K. “Hipotiroidi Hastalığı Teşhisinde Sınıflandırma Algoritmalarının Kullanımı”, Bilişim Teknolojileri Dergisi, vol.13, no.3, pp.255-268, 2020.
76. Bezek Güre, Ö. “Classification of Liver Disorders Diagnosis using Naïve Bayes Method”. Bitlis Eren Üniversitesi Fen Bilimleri Dergisi, 13(1), 153-160, 2024.
77. Aksu, G. & Dogan, N. “Comparison of Decision Trees Used in Data Mining= Veri madenciliginde kullanilan karar agaçlarinin karsilastirilmasi”. Pegem Journal of Education and Instruction, 9(4), 1183-1208, 2019.
78. Uğuz, E., Şahin, S. & Yılmaz, R. “PISA 2018 fen bilimleri puanlarının değerlendirilmesinde eğitsel veri madenciliğinin kullanımı”. Bilgi ve İletişim Teknolojileri Dergisi, 3(2), 212-227. 2021
79. Zeybekoğlu, Ş. & Koğar, H. “Investigation of variables explaining science literacy in PISA 2015 Turkey sample”. Journal of Measurement and Evaluation in Education and Psychology, 13(2), 145-163, 2022.
80. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W. ... & Liu, T. Y. “Lightgbm: A highly efficient gradient boosting decision tree". Advances in neural information processing systems, 30, 2017.
81. Wen, Z., He, B., Kotagiri, R., Lu, S. & Shi, J. Efficient gradient boosted decision tree training on GPUs. In 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS) (pp. 234-243). IEEE, 2018.