Performance of Classification Techniques on Smaller Group Prediction

Abstract

Classification techniques allow researchers to analyze data based on groups for the purposes of clustering or making predictions about group membership. Since there are many methods for utilizing classification analyses, such as Linear Discriminant Analysis (LDA), Logistic Regression (LR), and Classification and Regression Trees (CART), it is important to know which techniques perform better under which conditions to affect prediction accuracy. In the context of group prediction, it is crucial to consider the impact of group proportional sizes on prediction accuracy, particularly when comparing smaller groups to larger ones. This study evaluated the small group prediction accuracies of LDA, LR, and CART, controlling for number of groups, correlation, and number of predictor variables. Results showed that CART performed best in most cases for smaller and overall group prediction. In addition, a notable difference was observed in overall group prediction accuracy compared to small group prediction accuracy, with the overall group prediction accuracy being greater. Data conditions had a greater impact on LR and LDA than CART, and, in certain instances, LR showed superiority over the other two methods. The number of groups was the most influential factor on small group prediction, while the number of predictor variables, correlation, and method were of decreasing influence. In general, overall group prediction accuracy and small group prediction accuracy were negatively related. However, for the categories with an equal number of groups, the two were positively related.

Keywords

• method performance evaluation
• group membership
• classification accuracy
• simulation

References

1. Agresti, A. (2013). Categorical data analysis (3rd ed.). Wiley.
2. Barön, A. E. (1991). Misclassification among methods used for multiple group discrimination‐the effects of distributional properties. Statistics in Medicine, 10(5), 757-766. doi: https://doi.org/10.1002/sim.4780100511
3. Bates, B. E., Xie, D., Kwong, P. L., Kurichi, J. E., Ripley, D. C., & Stineman, M. G. (2014). One-year all-cause mortality after stroke: A prediction model. PM&R, 6(6), 473-483. doi: https://doi.org/10.1016/j.pmrj.2013.11.006
4. Bolin, J., & Finch, W. (2014). Supervised classification in the presence of misclassified training data: A Monte Carlo simulation study in the three-group case. Frontiers in Psychology, 5 doi:10.3389/fpsyg.2014.00118
5. Breiman, L., Friedman, J., Stone, C. J., & Olshen, R. A. (1984). Classification and regression trees. CRC Press.
6. Castonguay, A. C., Zoghi, Z., Zaidat, O. O., Burgess, R. E., Zaidi, S. F., Mueller‐Kronast, N., ... & Jumaa, M. A. (2023). Predicting functional outcome using 24‐hour post‐treatment characteristics: Application of machine learning algorithms in the STRATIS registry. Annals of Neurology, 93(1), 40-49. https://doi.org/10.1002/ana.26528
7. Chiang, Y. C. (2021). Evaluating the performance of classification and regression trees, random forests, and K-means clustering under controlled conditions (Doctoral dissertation, Indiana University).
8. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Earlbaum Associates.

9. Dattalo, P. (1995). A comparison of discriminant analysis and logistic regression. Journal of Social Service Research, 19(3-4), 121-144.
10. Dey, E. L., & Astin, A. W. (1993). Statistical alternatives for studying college student retention: A comparative analysis of logit, probit, and linear regression. Research in Higher Education, 34, 569-581. doi: https://doi.org/10.1007/BF00991920
11. Edwards, A. L. (1985). Experimental design in psychological research (5th ed.). NY: Harper & Row.
12. Fan, X., & Wang, L. (1999). Comparing linear discriminant function with logistic regression for the two-group classification problem. The Journal of Experimental Education, 67(3), 265-286. doi:10.1080/00220979909598356
13. Ferrer, A. J. A., & Wang, L. (1999). Comparing the Classification Accuracy among Nonparametric, Parametric Discriminant Analysis and Logistic Regression Methods (pp. 1-24, Rep.). Montreal: Paper presented at the Annual Meeting of the American Educational Research Association. https://eric.ed.gov/?id=ED432591
14. Finch, H. W., Bolin, J. E., & Kelley, K. (2014). Group membership prediction when known groups consist of unknown subgroups: A Monte Carlo comparison of methods. Frontiers in Psychology, 5. doi:10.3389/fpsyg.2014.00337
15. Finch, H., & Schneider, M. K. (2007). Classification accuracy of neural networks vs. discriminant analysis, logistic regression, and classification and regression trees: Three- and five-group cases. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 3(2), 47-57. doi:10.1027/1614-2241.3.2.47
16. Finch, W. H., & Schneider, M. K. (2006). Misclassification rates for four methods of group classification. Educational and Psychological Measurement, 66(2), 240-257. doi:10.1177/0013164405278579
17. Graf, R., Zeldovich, M., & Friedrich, S. (2023). Comparing linear discriminant analysis and supervised learning algorithms for binary classification—A method comparison study. Biometrical Journal, 66(1), 2200098. doi: https://doi.org/10.1002/bimj.202200098
18. Grassi, M., Villani, S., & Marinoni, A. (2001). Classification methods for the identification of “case” in epidemiological diagnosis of asthma. European Journal of Epidemiology, 17, 19-29. doi: https://doi.org/10.1023/A:1010987521885
19. Hassan, H. A., Hemdan, E. E. D., El‐Shafai, W., Shokair, M., & Abd El‐Samie, F. E. (2024). Detection of attacks on software defined networks using machine learning techniques and imbalanced data handling methods. Security and Privacy, 7(2), e350. doi: https://doi.org/10.1002/spy2.350
20. Hastie, T., Tibshirani R., Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer.
21. Hess, B., Olejnik, S., & Huberty, C. J. (2001). The efficacy of two improvement-over-chance effect sizes for two-group univariate comparisons under variance heterogeneity and nonnormality. Educational and Psychological Measurement, 61(6), 909-936. doi: https://doi.org/10.1177/00131640121971572
22. Hao, Z., Yang, B., Ruggiano, N., Ma, Y., Guo, Y., & Pan, X. (2022). Depression prediction amongst Chinese older adults with neurodegenerative diseases: A performance comparison between decision tree model and logistic regression analysis. The British Journal of Social Work, 52(1), 274-290. doi: https://doi.org/10.1093/bjsw/bcaa237
23. Hoang, M. L., Matrella, G., & Ciampolini, P. (2025). Metrological evaluation of contactless sleep position recognition using an accelerometric smart bed and machine learning. Sensors and Actuators A: Physical, 385, 116309. doi: https://doi.org/10.1016/j.sna.2025.116309
24. Holden, J. E., Finch, W. H., & Kelley, K. (2011). A comparison of two- group classification methods. Educational and Psychological Measurement, 71(5), 870-901. doi:10.1177/0013164411398357
25. Holden, J. E., & Kelley, K. (2010). The effects of initially misclassified data on the effectiveness of discriminant function analysis and finite mixture modeling. Educational and Psychological Measurement, 70(1), 36-55. doi:10.1177/0013164409344533
26. Huberty, C. J. (1994). Applied discriminant analysis. John Wiley & Sons.
27. Huberty, C. J., & Olejnik, S. (2006). Applied MANOVA and discriminant analysis (Vol. 498). John Wiley & Sons. IBM Corp. (2025). IBM SPSS Statistics for Windows, Version 30.0.0. Armonk, NY: IBM Corp. Retrieved from https://www.ibm.com/us-en/marketplace/statistical-analysis-and-reporting
28. Kiang, M. Y. (2003). A comparative assessment of classification methods. Decision Support Systems, 35(4), 441-454. doi: https://doi.org/10.1016/S0167-9236(02)00110-0
29. Lei, P., & Koehly, L. (2003). Linear discriminant analysis versus logistic regression: A comparison of classification errors in the two-group case. Journal of Experimental Education, 72(1), 25-49. doi: https://doi.org/10.1080/00220970309600878
30. Meshbane, A., & Morris, J. D. (1995). A method for selecting between linear and quadratic classification models in discriminant analysis. Journal of Experimental Education, 63(1), 263-273. doi: https://doi.org/10.1080/00220973.1995.9943813
31. Pai, D. R., Lawrence, K. D., Klimberg, R. K., & Lawrence, S. M. (2012). Analyzing the balancing of error rates for multi-group classification. Expert Systems with Applications, 39(17), 12869-12875. doi: https://doi.org/10.1016/j.eswa.2012.05.006
32. Paxton, P., Curran, P. J., Bollen, K. A., Kirby, J., & Chen, F. (2001). Monte Carlo Experiments: Design and implementation. Structural Equation Modeling, 8(2), 287-312. doi: https://doi.org/10.1207/S15328007SEM0802_7
33. Pierce, C. A., Block, R. A., & Aguinis, H. (2004). Cautionary note on reporting eta-squared values from multifactor ANOVA designs. Educational and Psychological Measurement, 64(6), 916-924. doi: https://doi.org/10.1177/0013164404264848
34. Pohar, M., Blas, M., & Turk, S. (2004). Comparison of logistic regression and linear discriminant analysis: A simulation study. Metodoloski Zvezki, 1(1), 143-161. http://mrvar.fdv.uni-lj.si/pub/mz/mz1.1/pohar.pdf
35. Preatoni, D. G., Nodari, M., Chirchella, R., Tosi, G., Wauters, L. A., & Martinoli, A. (2005). Identifying bats from time-expanded recordings of search calls: Comparing classification methods. Journal of Wildlife Management, 69(1), 1601-1614. doi: https://doi.org/10.2193/0022-541X(2005)69[1601:IBFTRO]2.0.CO;2
36. R Core Team (2016). R: A Language and Environment for Statistical Computing. R foundation for Statistical Computing, Vienna, Austria. Retrieved from https://www.r-project.org/
37. Rausch, J. R., & Kelley, K. (2009). A comparison of linear and mixture models for discriminant analysis under nonnormality. Behavior Research Methods, 41(1), 85-98. doi: https://doi.org/10.3758/BRM.41.1.85
38. Richardson, J. T. (2011). Eta squared and partial eta squared as measures of effect size in educational research. Educational Research Review, 6(2), 135-147. doi: https://doi.org/10.1016/j.edurev.2010.12.001
39. Ripley, B. D. (1994). Neural networks and related methods for classification. Journal of the Royal Statistical Society: Series B (Methodological), 3(1), 409-456. doi: https://doi.org/10.1111/j.2517-6161.1994.tb01990.x
40. Saboor, A., Usman, M., Ali, S., Samad, A., Abrar, M. F., & Ullah, N. (2022). A method for improving prediction of human heart disease using machine learning algorithms. Mobile Information Systems, 2022(1), 1410169. doi: https://doi.org/10.1155/2022/1410169
41. Sainani, K. L. (2014). Explanatory versus predictive modeling. PM&R, 6(9), 841-844. doi: https://doi.org/10.1016/j.pmrj.2014.08.941
42. Schumacher, M., Rossner, R., & Vach, W. (1996). Neural networks and logistic regression: Part I. Computational Statistics: Data Analysis, 21(1), 661-682. doi: https://doi.org/10.1016/0167-9473(95)00032-1
43. Selim, G. E. I., Hemdan, E. E. D., Shehata, A. M., & El-Fishawy, N. A. (2021). Anomaly events classification and detection system in critical industrial internet of things infrastructure using machine learning algorithms. Multimedia Tools and Applications, 80(8), 12619-12640. doi: https://doi.org/10.1007/s11042-020-10354-1
44. Soureshjani, M. H., & Kimiagari, A. M. (2013). Calculating the best cut off point using logistic regression and neural network on credit scoring problem-A case study of a commercial bank. African Journal of Business Management, 7(16), 1414. doi: 10.5897/AJBM11.394
45. Song, G., Ai, Z., Zhang, G., Peng, Y., Wang, W., & Yan, Y. (2022). Using machine learning algorithms to multidimensional analysis of subjective thermal comfort in a library. Building and Environment, 212, 108790. doi: https://doi.org/10.1016/j.buildenv.2022.108790
46. Suh, S., Suh, J., & Houston, I. (2007). Predictors of categorical at‐risk high school dropouts. Journal of Counseling & Development, 85(2), 196-203. doi: https://doi.org/10.1002/j.1556-6678.2007.tb00463.x
47. Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics. Boston: Pearson Education.
48. Williams, C. J., Lee, S. S., Fisher, R. A., & Dickerman, L. H. (1999). A comparison of statistical methods for prenatal screening for Down syndrome. Applied Stochastic Models in Business and Industry, 15(2), 89-101. doi: https://doi.org/10.1002/(SICI)1526-4025(199904/06)15:2<89::AID-ASMB366>3.0.CO;2-K
49. Zampogna, B., Torre, G., Zampoli, A., Parisi, F., Ferrini, A., Shanmugasundaram, S., ... & Papalia, R. (2024). Can machine learning predict the accuracy of preoperative planning for total hip arthroplasty, basing on patient-related factors? An explorative investigation on Supervised machine learning classification models. Journal of Clinical Orthopaedics and Trauma, 53, 102470. doi: https://doi.org/10.1016/j.jcot.2024.102470
50. Zavorka, S., & Perrett, J. J. (2014). Minimum sample size considerations for two-group linear and quadratic discriminant analysis with rare populations. Communications in Statistics-Simulation and Computation, 43(7), 1726-1739. doi: https://doi.org/10.1080/03610918.2012.744041