Entropy-Based Metrics for Evaluating Supervised Machine Learning Models: A Scoping Review

Mehmet Sıddık Çadırcı

doi:10.17776/csj.1933076

Entropy-Based Metrics for Evaluating Supervised Machine Learning Models: A Scoping Review

Abstract

In supervised machine learning, selecting appropriate performance metrics is crucial, as they directly influence model comparison, interpretation, and selection. Although traditional metrics such as accuracy, precision, recall, and F1 score are widely used, especially in complex or imbalanced classification tasks, they may not fully reflect prediction uncertainty or class-level variation. Thus, entropy-based metrics have emerged as complementary tools for evaluating uncertainty and information content in model outputs. This scoping review analyses 40 peer-reviewed empirical studies published between 2016 and 2024 that implement entropy-based metrics in supervised classification. The review determines the main entropy-based metrics used in the literature, examines how they are combined with or compared to traditional metrics, and emphasizes the contexts in which they provide added value. Entropy-based and information-theoretic measures can complement traditional classification metrics by providing additional information about uncertainty, calibration, distributional mismatch and class-level prediction behaviour, particularly in imbalanced or high-risk classification settings. Three barriers to wider adoption recur across the reviewed studies: computational cost, parameter sensitivity, and the absence of benchmarking standards.

Keywords

Classification, Entropy Metrics, Information Theory, Machine Learning Evaluation, Model Performance, Supervised Learning

References

Flach, P. (2019). Performance evaluation in machine learning: The good, the bad, the ugly, and the way forward. Proceedings of the AAAI Conference on Artificial Intelligence, 33(1), 9808–9814. https://doi.org/10.1609/aaai.v33i01.33019808
Dinga, R., Penninx, B. W. J. H., Veltman, D. J., Schmaal, L., & Marquand, A. F. (2019). Beyond accuracy: Measures for assessing machine learning models, pitfalls and guidelines [Preprint]. bioRxiv. https://doi.org/10.1101/743138
Azami, H., Faes, L., Escudero, J., Humeau-Heurtier, A., & Silva, L. E. V. (2023). Entropy analysis of univariate biomedical signals: Review and comparison of methods. In Frontiers in Entropy Across the Disciplines: Panorama of Entropy: Theory, Computation, and Applications (pp. 233–286). World Scientific. https://doi.org/10.1142/9789811259401_0009
Chernyshov, K. (2013). An information theoretic approach to constructing machine learning criteria. IFAC Proceedings Volumes, 46(11), 269–274. https://doi.org/10.3182/20130703-3-FR-4038.00145
Li, J., Izakian, H., Pedrycz, W., & Jamal, I. (2021). Clustering-based anomaly detection in multivariate time series data. Applied Soft Computing, 100, 106919. https://doi.org/10.1016/j.asoc.2020.106919
Fernandes, M., Corchado, J. M., & Marreiros, G. (2022). Machine learning techniques applied to mechanical fault diagnosis and fault prognosis in the context of real industrial manufacturing use-cases: A systematic literature review. Applied Intelligence, 52(12), 14246–14280. https://doi.org/10.1007/s10489-022-03344-3
Ahsan, M. M., & Siddique, Z. (2022). Machine learning-based heart disease diagnosis: A systematic literature review. Artificial Intelligence in Medicine, 128, 102289. https://doi.org/10.1016/j.artmed.2022.102289
Madukaife, M. S., & Phuc, H. D. (2024). Estimation of Shannon differential entropy: An extensive comparative review [Preprint]. arXiv. https://arxiv.org/abs/2406.19432
Lavesson, N., & Davidsson, P. (2007). Evaluating learning algorithms and classifiers. International Journal of Intelligent Information and Database Systems, 1(1), 37–52. https://doi.org/10.1504/IJIIDS.2007.013284
Stąpor, K. (2018). Evaluating and comparing classifiers: Review, some recommendations and limitations. In Proceedings of the 10th International Conference on Computer Recognition Systems CORES 2017 (pp. 12–21). Springer. https://doi.org/10.1007/978-3-319-59162-9_2

Grandini, M., Bagli, E., & Visani, G. (2020). Metrics for multi-class classification: An overview [Preprint]. arXiv. https://arxiv.org/abs/2008.05756
Golan, A. (2017). Foundations of info-metrics: Modeling, inference, and imperfect information. Oxford University Press. https://doi.org/10.1093/oso/9780199349524.001.0001
Carbonneau, M.-A., Cheplygina, V., Granger, E., & Gagnon, G. (2018). Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition, 77, 329–353. https://doi.org/10.1016/j.patcog.2017.10.009
Cadirci, M. S., & Singull, M. (2026). A new estimator of Kullback–Leibler divergence via Shannon entropy. Entropy, 28(7), 720. https://doi.org/10.3390/e28070720
Erickson, B. J., & Kitamura, F. (2021). Magician's corner: 9. Performance metrics for machine learning models. Radiology: Artificial Intelligence, 3(3), e200126. https://doi.org/10.1148/ryai.2021200126
Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009). Systematic literature reviews in software engineering – A systematic literature review. Information and Software Technology, 51(1), 7–15. https://doi.org/10.1016/j.infsof.2008.09.009
Opitz, D., & Maclin, R. (1999). Popular ensemble methods: An empirical study. Journal of Artificial Intelligence Research, 11, 169–198. https://doi.org/10.1613/jair.614
Japkowicz, N., & Stephen, S. (2002). The class imbalance problem: A systematic study. Intelligent Data Analysis, 6(5), 429–449. https://doi.org/10.3233/IDA-2002-6504
Raschka, S. (2018). Model evaluation, model selection, and algorithm selection in machine learning [Preprint]. arXiv. https://arxiv.org/abs/1811.12808
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018
Schölkopf, B., & Smola, A. J. (2002). Learning with kernels: Support vector machines, regularization, optimization, and beyond. MIT Press. https://doi.org/10.7551/mitpress/4175.001.0001
Hsu, C.-W., Chang, C.-C., & Lin, C.-J. (2003). A practical guide to support vector classification (Technical report). Department of Computer Science, National Taiwan University, Taipei.
Biggio, B., & Roli, F. (2018). Wild patterns: Ten years after the rise of adversarial machine learning. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (pp. 2154–2156). ACM. https://doi.org/10.1145/3243734.3264418
Faust, O., Hagiwara, Y., Hong, T. J., Lih, O. S., & Acharya, U. R. (2018). Deep learning for healthcare applications based on physiological signals: A review. Computer Methods and Programs in Biomedicine, 161, 1–13. https://doi.org/10.1016/j.cmpb.2018.04.005
Lee, J.-G., Jun, S., Cho, Y.-W., Lee, H., Kim, G. B., Seo, J. B., & Kim, N. (2017). Deep learning in medical imaging: General overview. Korean Journal of Radiology, 18(4), 570–584. https://doi.org/10.3348/kjr.2017.18.4.570
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770–778). IEEE. https://doi.org/10.1109/CVPR.2016.90
Mijwil, M. M., & Aggarwal, K. (2022). A diagnostic testing for people with appendicitis using machine learning techniques. Multimedia Tools and Applications, 81(5), 7011–7023. https://doi.org/10.1007/s11042-022-11939-8
Musunuru, S., Paranjape, J. N., Dubey, R. K., & Venkoparao, V. G. (2020). Characterizing the weight space for different learning models. In 2020 International Conference on Computing and Data Science (CDS) (pp. 109–113). IEEE. https://doi.org/10.1109/CDS49703.2020.00028
Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324
Liaw, A., & Wiener, M. (2002). Classification and regression by randomForest. R News, 2(3), 18–22.
Budianti, L., & Suliadi. (2022). Metode weighted random forest dalam klasifikasi prediksi kelangsungan hidup pasien gagal jantung. Bandung Conference Series: Statistics, 2(2), 103–110. https://doi.org/10.29313/bcss.v2i2.3318
Bahrawi, N. (2019). Sentiment analysis using random forest algorithm-online social media based. Journal of Information Technology and Its Utilization, 2(2), 29–33. https://doi.org/10.30818/jitu.2.2.2695
Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1(1), 81–106. https://doi.org/10.1007/BF00116251
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017). Classification and regression trees. Routledge. https://doi.org/10.1201/9781315139470
Liu, H., & Motoda, H. (1998). Feature selection for knowledge discovery and data mining. Springer. https://doi.org/10.1007/978-1-4615-5689-3
Hosmer, D. W., Jr., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression (3rd ed.). John Wiley & Sons. https://doi.org/10.1002/9781118548387
Ahn, I., Gwon, H., Kang, H., Kim, Y., Seo, H., Choi, H., Cho, H. N., Kim, M., Jun, T. J., & Kim, Y.-H. (2021). Machine learning–based hospital discharge prediction for patients with cardiovascular diseases: Development and usability study. JMIR Medical Informatics, 9(11), e32662. https://doi.org/10.2196/32662
Fitrianah, D., Dwiasnati, S., Hikmayanti, H., & Baihaqi, K. A. (2021). Penerapan metode machine learning untuk prediksi nasabah potensial menggunakan algoritma klasifikasi naïve bayes. Faktor Exacta, 14(2), 92–99. https://doi.org/10.30998/faktorexacta.v14i2.9297
McCallum, A., & Nigam, K. (1998). A comparison of event models for naive Bayes text classification. In AAAI-98 Workshop on Learning for Text Categorization (pp. 41–48). AAAI Press.
Wibawa, A. P., Kurniawan, A. C., Murti, D. M. P., Adiperkasa, R. P., Putra, S. M., Kurniawan, S. A., & Nugraha, Y. R. (2019). Naïve Bayes classifier for journal quartile classification. International Journal of Recent Contributions from Engineering, Science & IT (iJES), 7(2), 91–99. https://doi.org/10.3991/ijes.v7i2.10659
Cover, T., & Hart, P. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21–27. https://doi.org/10.1109/TIT.1967.1053964
Cadirci, M. S., Evans, D., Leonenko, N., & Makogin, V. (2022). Entropy-based test for generalized Gaussian distributions. Computational Statistics & Data Analysis, 173, 107502. https://doi.org/10.1016/j.csda.2022.107502
Altman, N. S. (1992). An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician, 46(3), 175–185. https://doi.org/10.1080/00031305.1992.10475879
Cadirci, M. S. (2025). Non-parametric goodness-of-fit tests using Tsallis entropy measures. Entropy, 27(12), 1210. https://doi.org/10.3390/e27121210
Singhal, A., & Sharma, D. K. (2021). Entropy analysis of a dataset using machine learning approach. In Innovations in Information and Communication Technology (pp. 1–6). https://doi.org/10.46532/978-81-950008-7-6_001
Dwivedi, Y. K., Rana, N. P., Slade, E. L., Williams, M. D., & Weerakkody, V. (2017). Predicting the "helpfulness" of online consumer reviews. Journal of Business Research, 70, 346–355. https://doi.org/10.1016/j.jbusres.2016.08.008
Wang, J., Sun, Q., & Zhou, C. (2023). Insider threat detection based on deep clustering of multi-source behavioral events. Applied Sciences, 13(24), 13021. https://doi.org/10.3390/app132413021
Lipitakis, E., & Lipitakis, E. (2014). On machine learning with imbalanced data: Evaluation metrics and performance. In 2014 International Conference on Computational Science and Computational Intelligence (CSCI). IEEE. https://doi.org/10.1109/CSCI.2014.81
Carbonneau, M.-A., Zaidi, J., Boilard, J., & Gagnon, G. (2024). Measuring disentanglement: A review of metrics. IEEE Transactions on Neural Networks and Learning Systems, 35(7), 8747–8761. https://doi.org/10.1109/TNNLS.2022.3218982
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023). From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable AI. ACM Computing Surveys, 55(13s), Article 295, 1–42. https://doi.org/10.1145/3583558
Dougherty, J., Kohavi, R., & Sahami, M. (1995). Supervised and unsupervised discretization of continuous features. In Proceedings of the 12th International Conference on Machine Learning (pp. 194–202). Morgan Kaufmann. https://doi.org/10.1016/B978-1-55860-377-6.50032-3
Dubnov, Y. A. (2019). Entropy-based estimation in classification problems. Automation and Remote Control, 80(3), 502–512. https://doi.org/10.1134/S0005117919030093
Sharma, V., & Dey, S. (2012). A comparative study of feature selection and machine learning techniques for sentiment analysis. In Proceedings of the ACM Research in Adaptive and Convergent Systems (pp. 1–7). ACM. https://doi.org/10.1145/2401603.2401605
Devi, K. L., Subathra, P., & Kumar, P. N. (2016). Performance evaluation of sentiment classification using query strategies in a pool-based active learning scenario. In Computational Intelligence and Networks (pp. 63–73). Springer. https://doi.org/10.1007/978-981-10-0251-9_8
Mayer, C., & Timofte, R. (2020). Adversarial sampling for active learning. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) (pp. 3060–3068). IEEE. https://doi.org/10.1109/WACV45572.2020.9093556

Details

Primary Language

English

Subjects

Computational Statistics, Statistical Theory, Theory of Sampling

Journal Section

Review

Authors

Mehmet Sıddık Çadırcı ^*
0000-0001-7654-7609
Türkiye

Publication Date

October 27, 2026

Submission Date

April 18, 2026

Acceptance Date

September 8, 2026

Published in Issue

Year 2026 Volume: 47 Number: 5

DOI

https://doi.org/10.17776/csj.1933076

IZ

https://izlik.org/JA84UM98CY

APA

Çadırcı, M. S. (2026). Entropy-Based Metrics for Evaluating Supervised Machine Learning Models: A Scoping Review. Cumhuriyet Science Journal, 47(5), 1041-1051. https://doi.org/10.17776/csj.1933076

AMA

1.Çadırcı MS. Entropy-Based Metrics for Evaluating Supervised Machine Learning Models: A Scoping Review. CSJ. 2026;47(5):1041-1051. doi:10.17776/csj.1933076

Chicago

Çadırcı, Mehmet Sıddık. 2026. “Entropy-Based Metrics for Evaluating Supervised Machine Learning Models: A Scoping Review”. Cumhuriyet Science Journal 47 (5): 1041-51. https://doi.org/10.17776/csj.1933076.

EndNote

Çadırcı MS (October 1, 2026) Entropy-Based Metrics for Evaluating Supervised Machine Learning Models: A Scoping Review. Cumhuriyet Science Journal 47 5 1041–1051.

IEEE

[1]M. S. Çadırcı, “Entropy-Based Metrics for Evaluating Supervised Machine Learning Models: A Scoping Review”, CSJ, vol. 47, no. 5, pp. 1041–1051, Oct. 2026, doi: 10.17776/csj.1933076.

ISNAD

Çadırcı, Mehmet Sıddık. “Entropy-Based Metrics for Evaluating Supervised Machine Learning Models: A Scoping Review”. Cumhuriyet Science Journal 47/5 (October 1, 2026): 1041-1051. https://doi.org/10.17776/csj.1933076.

JAMA

1.Çadırcı MS. Entropy-Based Metrics for Evaluating Supervised Machine Learning Models: A Scoping Review. CSJ. 2026;47:1041–1051.

MLA

Çadırcı, Mehmet Sıddık. “Entropy-Based Metrics for Evaluating Supervised Machine Learning Models: A Scoping Review”. Cumhuriyet Science Journal, vol. 47, no. 5, Oct. 2026, pp. 1041-5, doi:10.17776/csj.1933076.

Vancouver

1.Mehmet Sıddık Çadırcı. Entropy-Based Metrics for Evaluating Supervised Machine Learning Models: A Scoping Review. CSJ. 2026 Oct. 1;47(5):1041-5. doi:10.17776/csj.1933076

As of 2026, Cumhuriyet Science Journal will be published in six issues per year, released in February, April, June, August, October, and December

Entropy-Based Metrics for Evaluating Supervised Machine Learning Models: A Scoping Review

Abstract

Keywords

References

Details

Primary Language

Subjects

Journal Section

Authors

Publication Date

Submission Date

Acceptance Date

Published in Issue

DOI

IZ

Cite