A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets

Necati Vardar; Mehmet Fatih Ören

doi:10.35377/saucis...1833058

A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets

Abstract

Class imbalance remains a critical challenge in supervised learning, often biasing classifiers toward majority classes. While resampling techniques like Synthetic Minority Oversampling Technique (SMOTE) are widely used, the combined effect of data balancing and hyperparameter optimization across diverse datasets is rarely systematically explored. This study presents a comprehensive comparative analysis of four classification algorithms—Naive Bayes (NB), K-Nearest Neighbors (K-NN), Artificial Neural Networks (ANN), and Random Forest (RF)—across ten benchmark datasets from the UCI Machine Learning Repository. Unlike previous studies relying on default parameters, this research employs a rigorous Grid Search strategy to optimize hyperparameters for each algorithm within a rigorous SMOTE-balanced stratified cross-validation pipeline to ensure robust evaluation. Performance was assessed using a wide range of metrics, including Accuracy, Precision, Recall, F1-score, and Area Under the Curve (AUC). Experimental results reveal that ANN achieved the highest robustness in high-dimensional and complex categorical datasets (e.g., Car Evaluation F1-score: 0.990), significantly outperforming traditional models. Conversely, RF demonstrated superior stability in datasets with high feature dimensionality (e.g., Arrhythmia F1-score: 0.600) and chemical interactions (e.g., QSAR Fish Toxicity F1-score: 0.839). While K-NN remained competitive in low-dimensional spaces, NB struggled with complex feature dependencies. This study contributes to the literature by demonstrating that algorithmic superiority is context-dependent and providing a data-driven framework for selecting classifiers based on structural characteristics such as dimensionality, categorical complexity, and sample size.

Keywords

Ethical Statement

It is declared that during the preparation process of this study, scientific and ethical principles were followed, and all the sources benefited from are stated in the bibliography. Since this study utilizes public datasets, no formal ethics committee approval was required.

Thanks

The authors thank KTO Karatay University for the research environment.

References

A. Tharwat, “Classification assessment methods,” Appl. Comput. Informat., vol. 17, no. 1, pp. 168–192, 2021, doi: 10.1016/j.aci.2018.08.003. [Online]. Available: https://doi.org/10.1016/j.aci.2018.08.003
T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets,” PLOS ONE, vol. 10, no. 3, Art. no. e0118432, 2015, doi: 10.1371/journal.pone.0118432. [Online]. Available: https://doi.org/10.1371/journal.pone.0118432
N. V. Chawla, N. Japkowicz, and A. Kotcz, “Editorial: Special issue on learning from imbalanced data sets,” ACM SIGKDD Explor. Newslett., vol. 6, no. 1, pp. 1–6, 2004, doi: 10.1145/1007730.1007733. [Online]. Available: https://doi.org/10.1145/1007730.1007733
J. M. Johnson and T. M. Khoshgoftaar, “Survey on deep learning with class imbalance,” J. Big Data, vol. 6, no. 1, Art. no. 27, 2019, doi: 10.1186/s40537-019-0192-5. [Online]. Available: https://doi.org/10.1186/s40537-019-0192-5
W. Chen, K. Yang, Z. Yu, Y. Shi, and C. L. P. Chen, “A survey on imbalanced learning: Latest research, applications and future directions,” Artif. Intell. Rev., vol. 57, Art. no. 137, 2024, doi: 10.1007/s10462-024-10759-6. [Online]. Available: https://doi.org/10.1007/s10462-024-10759-6
M. Salmi et al., “Handling imbalanced medical datasets: Review of a decade of research,” Artif. Intell. Rev., vol. 57, Art. no. 273, 2024, doi: 10.1007/s10462-024-10884-2. [Online]. Available: https://doi.org/10.1007/s10462-024-10884-2
H. Kaur, H. S. Pannu, and A. K. Malhi, “A systematic review on imbalanced data challenges in machine learning: Applications and solutions,” ACM Comput. Surv., vol. 52, no. 4, pp. 1–36, 2019, doi: 10.1145/3343440. [Online]. Available: https://doi.org/10.1145/3343440
N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic minority over-sampling technique,” J. Artif. Intell. Res., vol. 16, pp. 321–357, 2002, doi: 10.1613/jair.953. [Online]. Available: https://doi.org/10.1613/jair.953

A. Fernández, S. García, M. Galar, R. C. Prati, B. Krawczyk, and F. Herrera, Learning from Imbalanced Data Sets. Cham, Switzerland: Springer, 2018, doi: 10.1007/978-3-319-98074-4. [Online]. Available: https://doi.org/10.1007/978-3-319-98074-4
D. Dua and C. Graff, “UCI machine learning repository,” School Inf. Comput. Sci., Univ. California, Irvine, CA, USA, 2019. [Online]. Available: https://archive.ics.uci.edu
G. James, D. Witten, T. Hastie, and R. Tibshirani, An Introduction to Statistical Learning: With Applications in R. New York, NY, USA: Springer, 2013.
H. Zhang, “The optimality of Naive Bayes,” in Proc. 17th Int. Florida Artif. Intell. Res. Soc. Conf. (FLAIRS), Menlo Park, CA, USA: AAAI Press, 2004, pp. 562–567.
I. Rish, “An empirical study of the Naive Bayes classifier,” in Proc. IJCAI Workshop Empirical Methods Artif. Intell., vol. 3, no. 22, 2001, pp. 41–46.
J. Han, M. Kamber, and J. Pei, Data Mining: Concepts and Techniques, 3rd ed. Waltham, MA, USA: Morgan Kaufmann, 2012.
T. Cover and P. Hart, “Nearest neighbor pattern classification,” IEEE Trans. Inf. Theory, vol. IT-13, no. 1, pp. 21–27, Jan. 1967.
S. Haykin, Neural Networks: A Comprehensive Foundation, 2nd ed. Englewood Cliffs, NJ, USA: Prentice-Hall, 1999.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, no. 6088, pp. 533–536, 1986.
L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, 2001.
C. Chen, A. Liaw, and L. Breiman, “Using random forest to learn imbalanced data,” Dept. Statist., Univ. California, Berkeley, CA, USA, Tech. Rep., 2004.
A. A. Khan, O. Chaudhari, and R. Chandra, “A review of ensemble learning and data augmentation models for class imbalanced problems: Combination, implementation and evaluation,” Expert Syst. Appl., vol. 244, Art. no. 122778, 2024, doi: 10.1016/j.eswa.2023.122778. [Online]. Available: https://doi.org/10.1016/j.eswa.2023.122778
G. E. A. P. A. Batista, R. C. Prati, and M. C. Monard, “A study of the behavior of several methods for balancing machine learning training data,” ACM SIGKDD Explor. Newslett., vol. 6, no. 1, pp. 20–29, 2004, doi: 10.1145/1007730.1007735. [Online]. Available: https://doi.org/10.1145/1007730.1007735
H. Guvenir, B. Acar, H. Muderrisoglu, and R. Quinlan, “Arrhythmia,” UCI Machine Learning Repository, 1997, doi: 10.24432/C5BS32. [Online]. Available: https://doi.org/10.24432/C5BS32
R. Fisher, “Iris,” UCI Machine Learning Repository, 1936, doi: 10.24432/C56C76. [Online]. Available: https://doi.org/10.24432/C56C76
M. Bohanec, “Car evaluation,” UCI Machine Learning Repository, 1988, doi: 10.24432/C5JP48. [Online]. Available: https://doi.org/10.24432/C5JP48
“Labor relations,” UCI Machine Learning Repository, 1988, doi: 10.24432/C5CP4Q. [Online]. Available: https://doi.org/10.24432/C5CP4Q
“Mushroom,” UCI Machine Learning Repository, 1981, doi: 10.24432/C5959T. [Online]. Available: https://doi.org/10.24432/C5959T
Ş. Gül and F. Rahim, “Period changer,” UCI Machine Learning Repository, 2021, doi: 10.24432/C5B31D. [Online]. Available: https://doi.org/10.24432/C5B31D
“Rice (Cammeo and Osmancik),” UCI Machine Learning Repository, 2019, doi: 10.24432/C5MW4Z. [Online]. Available: https://doi.org/10.24432/C5MW4Z
D. Ballabio, M. Cassotti, V. Consonni, and R. Todeschini, “QSAR fish toxicity,” UCI Machine Learning Repository, 2015, doi: 10.24432/C5JG7B. [Online]. Available: https://doi.org/10.24432/C5JG7B
E. Fokoue and N. Gunduz, “Türkiye student evaluation,” UCI Machine Learning Repository, 2013, doi: 10.24432/C5S02S. [Online]. Available: https://doi.org/10.24432/C5S02S
S. Aeberhard and M. Forina, “Wine,” UCI Machine Learning Repository, 1992, doi: 10.24432/C5PC7J. [Online]. Available: https://doi.org/10.24432/C5PC7J
B. Bischl et al., “Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges,” WIREs Data Mining Knowl. Discovery, vol. 13, no. 2, Art. no. e1484, 2023, doi: 10.1002/widm.1484. [Online]. Available: https://doi.org/10.1002/widm.1484
P. Thölke et al., “Class imbalance should not throw you off balance: Choosing the right classifiers and performance metrics for brain decoding with imbalanced data,” NeuroImage, vol. 277, Art. no. 120253, 2023, doi: 10.1016/j.neuroimage.2023.120253. [Online]. Available: https://doi.org/10.1016/j.neuroimage.2023.120253
D. H. Wolpert, “The lack of a priori distinctions between learning algorithms,” Neural Comput., vol. 8, no. 7, pp. 1341–1390, 1996, doi: 10.1162/neco.1996.8.7.1341. [Online]. Available: https://doi.org/10.1162/neco.1996.8.7.1341
V. Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelczyk, and G. Kasneci, “Deep neural networks and tabular data: A survey,” IEEE Trans. Neural Netw. Learn. Syst., vol. 35, no. 6, pp. 7499–7519, Jun. 2024, doi: 10.1109/TNNLS.2022.3229161. [Online]. Available: https://doi.org/10.1109/TNNLS.2022.3229161
H. He and E. A. Garcia, “Learning from imbalanced data,” IEEE Trans. Knowl. Data Eng., vol. 21, no. 9, pp. 1263–1284, Sep. 2009, doi: 10.1109/TKDE.2008.239. [Online]. Available: https://doi.org/10.1109/TKDE.2008.239
A. K. Azlim Khan and N. H. Ahamed Hassain Malim, “Comparative studies on resampling techniques in machine learning and deep learning models for drug-target interaction prediction,” Molecules, vol. 28, no. 4, Art. no. 1663, 2023, doi: 10.3390/molecules28041663. [Online]. Available: https://doi.org/10.3390/molecules28041663

Details

Primary Language

English

Subjects

Artificial Intelligence (Other)

Journal Section

Research Article

Authors

Necati Vardar ^*
0000-0002-5017-9788
Türkiye

Mehmet Fatih Ören
0009-0001-0683-0000
Türkiye

Publication Date

September 30, 2026

Submission Date

December 1, 2025

Acceptance Date

February 3, 2026

Published in Issue

Year 2026 Volume: 9 Number: 4

DOI

https://doi.org/10.35377/saucis...1833058

IZ

https://izlik.org/JA32FL92XJ

Cite

RIS / Bibtex

APA

Vardar, N., & Ören, M. F. (2026). A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets. Sakarya University Journal of Computer and Information Sciences, 9(4), 1051-1061. https://doi.org/10.35377/saucis...1833058

AMA

1.Vardar N, Ören MF. A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets. SAUCIS. 2026;9(4):1051-1061. doi:10.35377/saucis.1833058

Chicago

Vardar, Necati, and Mehmet Fatih Ören. 2026. “A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets”. Sakarya University Journal of Computer and Information Sciences 9 (4): 1051-61. https://doi.org/10.35377/saucis. 1833058.

EndNote

Vardar N, Ören MF (September 1, 2026) A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets. Sakarya University Journal of Computer and Information Sciences 9 4 1051–1061.

IEEE

[1]N. Vardar and M. F. Ören, “A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets”, SAUCIS, vol. 9, no. 4, pp. 1051–1061, Sept. 2026, doi: 10.35377/saucis...1833058.

ISNAD

Vardar, Necati - Ören, Mehmet Fatih. “A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets”. Sakarya University Journal of Computer and Information Sciences 9/4 (September 1, 2026): 1051-1061. https://doi.org/10.35377/saucis. 1833058.

JAMA

1.Vardar N, Ören MF. A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets. SAUCIS. 2026;9:1051–1061.

MLA

Vardar, Necati, and Mehmet Fatih Ören. “A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets”. Sakarya University Journal of Computer and Information Sciences, vol. 9, no. 4, Sept. 2026, pp. 1051-6, doi:10.35377/saucis. 1833058.

Vancouver

1.Necati Vardar, Mehmet Fatih Ören. A Comparative Performance Evaluation of Classification Algorithms on Imbalanced Datasets. SAUCIS. 2026 Sep. 1;9(4):1051-6. doi:10.35377/saucis. 1833058