A Comparative Study of Data Balancing Algorithms to Examine Optimal Class Distribution in Imbalanced Datasets: A Simulation Based Approach

Abstract

In a two-class dataset, the class imbalance problem arises if there is a considerable difference between the number of samples in the classes. Many data balancing algorithms have been proposed to address this issue. However, only a limited number of studies have examined the candidate balance ratios of certain balancing algorithms, often focusing on real datasets. Unlike previous studies, this research evaluates seven balancing algorithms in terms of their predictive performance for an estimated population parameter (EP) and examines minority-majority class distributions yielding performance comparable to EP using an original simulation scenario. In this study, imbalanced datasets were sampled from a simulated population dataset and gradually balanced using random oversampling (ROS), synthetic minority oversampling technique (SMOTE), majority weighted minority oversampling technique (MWMOTE), adaptive synthetic sampling approach (ADASYN), random undersampling (RUS), random under boosting (RUSBoost), and under bagging (UB) algorithms. The classification and regression trees (CART) method was used to classify the data at each step, and the area under the ROC curve (AUC) was employed to evaluate the performance of the balancing algorithms. The findings obtained under the present simulation setting indicate that RUSBoost and UB algorithms yield statistically higher results than EP when certain balance ratios are exceeded. Meanwhile, within the evaluated simulation setting and the CART–AUC framework, other methods do not surpass EP and generally achieve their highest mean AUC values at full balance (50:50).

Keywords

• Imbalanced data
• Balancing algorithms
• Classification and regression trees
• Simulation

References

1. Barua, S., Islam, M.M., Yao, X., and Murase, K., "MWMOTE-majority weighted minority oversampling technique for imbalanced data set learning", IEEE Transactions on Knowledge and Data Engineering, 26(2): 405-425, (2012). DOI: https://doi.org/10.1109/TKDE.2012.232
2. Chawla, N.V., Bowyer, K.W., Hall, L.O., and Kegelmeyer, W.P., "SMOTE: synthetic minority over-sampling technique", Journal of Artificial Intelligence Research, 16: 321-357, (2002). DOI: https://doi.org/10.1613/jair.953
3. Fernández, A., García, S., Galar, M., Prati, R.C., Krawczyk, B., and Herrera, F., "Learning from imbalanced data sets", Springer, (2018). DOI: https://doi.org/10.1007/978-3-319-98074-4
4. He, H., Bai, Y., Garcia, E.A., and Li, S., "ADASYN: Adaptive synthetic sampling approach for imbalanced learning", 2008 IEEE International Joint Conference on Neural Networks, Hong Kong, China, 1322-1328, (2008). DOI: https://doi.org/10.1109/IJCNN.2008.4633969
5. Santos, M.S., Soares, J.P., Abreu, P.H., Araujo, H., and Santos, J., "Cross-validation for imbalanced datasets: Avoiding overoptimistic and overfitting approaches", IEEE Computational Intelligence Magazine, 13(4): 59-76, (2018). DOI: https://doi.org/10.1109/MCI.2018.2866730
6. Seiffert, C., Khoshgoftaar, T.M., Van Hulse, J., and Napolitano, A., "RUSBoost: A hybrid approach to alleviating class imbalance", IEEE Transactions on Systems, Man, Cybernetics-Part A: Systems Humans, 40(1): 185-197, (2009). DOI: https://doi.org/10.1109/TSMCA.2009.2029559
7. Sun, Z., Song, Q., Zhu, X., Sun, H., Xu, B., and Zhou, Y., "A novel ensemble method for classifying imbalanced data", Pattern Recognition, 48(5): 1623-1637, (2015). DOI: https://doi.org/10.1016/j.patcog.2014.11.014
8. Galar, M., Fernandez, A., Barrenechea, E., Bustince, H., and Herrera, F., "A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches", IEEE Transactions on Systems, Man, and Cybernetics Part C: Applications and Reviews, 42(4):463-484, (2011). DOI: https://doi.org/10.1109/TSMCC.2011.2161285

9. Hasanin, T., and Khoshgoftaar, T., "The effects of random undersampling with simulated class imbalance for big data", 2018 IEEE International Conference on Information Reuse and Integration, 70-79, (2018). DOI: https://doi.org/10.1109/IRI.2018.00018
10. Japkowicz, N., "The class imbalance problem: Significance and strategies", Proceedings of the International Conference on Artificial Intelligence, (2000).
11. Kamei, Y., Monden, A., Matsumoto, S., Kakimoto, T., and Matsumoto, K., "The effects of over and under sampling on fault-prone module detection", First International Symposium on Empirical Software Engineering and Measurement, Madrid, Spain, 196-204, (2007). DOI: https://doi.org/10.1109/ESEM.2007.28
12. López, V., Fernández, A., García, S., Palade, V., and Herrera, F., "An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics", Information Sciences, 250: 113-141, (2013). DOI: https://doi.org/10.1016/j.ins.2013.07.007
13. Tyagi, S., and Mittal, S., “Sampling approaches for imbalanced data classification problem in machine learning”, Proceedings of ICRIC 2019, Springer, 209-221, (2020). DOI: https://doi.org/10.1007/978-3-030-29407-6_17
14. Van Hulse, J., Khoshgoftaar, T.M., and Napolitano, A., "Experimental perspectives on learning from imbalanced data", Proceedings of the 24th International Conference on Machine Learning, 935-942, (2007). DOI: https://doi.org/10.1145/1273496.1273614
15. Batista, G.E., Prati, R.C., and Monard, M.C., "A study of the behavior of several methods for balancing machine learning training data", ACM SIGKDD Explorations Newsletter, 6(1): 20-29, (2004). DOI: https://doi.org/10.1145/1007730.1007735
16. Weiss, G.M., and Provost, F., "Learning when training data are costly: The effect of class distribution on tree induction", Journal of Artificial Intelligence Research, 19: 315-354, (2003). DOI: https://doi.org/10.1613/jair.1199
17. Khoshgoftaar, T.M., Seiffert, C., Van Hulse, J., Napolitano, A., and Folleco A., "Learning with limited minority class data", Sixth International Conference on Machine Learning and Applications, 348-353, (2007). DOI: https://doi.org/10.1109/ICMLA.2007.76
18. Albisua, I., Arbelaitz, O., Gurrutxaga, I., Lasarguren, A., Muguerza, J., and Pérez, J.M., "The quest for the optimal class distribution: an approach for enhancing the effectiveness of learning via resampling methods for imbalanced data sets", Progress in Artificial Intelligence, 2(1): 45-63, (2013). DOI: https://doi.org/10.1007/s13748-012-0034-6
19. Barandela, R., Valdovinos, R.M., and Sánchez, J.S., "New applications of ensembles of classifiers", Pattern Analysis and Applications, 6(3): 245-256, (2003). DOI: https://doi.org/10.1007/s10044-003-0192-z
20. Bujang, M.A., and Adnan, T.H., "Requirements for minimum sample size for sensitivity and specificity analysis", Journal of Clinical and Diagnostic Research, 10(10): YE01-YE6, (2016). DOI: https://doi.org/10.7860/JCDR/2016/18129.8744
21. Amin, A., Anwar, S., Adnan, A., Nawaz, M., Howard, N., Qadir, J., Hawalah, A., and Hussain, A., "Comparing oversampling techniques to handle the class imbalance problem: A customer churn prediction case study", IEEE Access, 4: 7940-7957, (2016). DOI: https://doi.org/10.1109/ACCESS.2016.2619719
22. Bennin, K.E., Keung, J., Monden, A., Kamei, Y., and Ubayashi, N., "Investigating the effects of balanced training and testing datasets on effort-aware fault prediction models", 2016 IEEE 40th Annual Computer Software and Applications Conference, 154-163, (2016). DOI: https://doi.org/10.1109/COMPSAC.2016.144
23. Chen, S., He, H., and Garcia, E.A., "RAMOBoost: Ranked minority oversampling in boosting", IEEE Transactions on Neural Networks, 21(10): 1624-1642, (2010). DOI: https://doi.org/10.1109/TNN.2010.2066988
24. Rashu, R.I., Haq, N., and Rahman, R.M., "Data mining approaches to predict final grade by overcoming class imbalance problem", 2014 17th International Conference on Computer and Information Technology, 14-19, (2014). DOI: https://doi.org/10.1109/ICCITechn.2014.7073095