Model-Based Feature Selection Using Structural Equation Modeling for Enhanced Classification Performance in High-Dimensional Datasets

Abstract

Feature selection is becoming more and more important for machine learning and data mining. Especially for high dimensional datasets, it is necessary to filter out irrelevant and unnecessary features to overcome the problems of overfitting and multidimensionality. We hypothesized that an effective feature selection can be made with a model-based approach using the Structural Equation Modeling (SEM) method. The dataset consists of 2969 samples and 117 features. First, a measurement model created was tested with confirmatory factor analysis (CFA) and the number of features was reduced to 58 by removing the statistically insignificant features. In SEM analysis, sub-feature sets consisting of 55, 52, 41 and 35 features were obtained by removing the variables whose relationship was below the threshold values determined for the standardized regression coefficient (SRC). The obtained sub-feature sets were tested with a multilayer perceptron (MLP) and their effect on performance was examined. Results were compared with random forest feature importance as baseline method. SEM and random forest have generally performed very closely. While sub-feature sets created with the random forest in two-class classification produced better results, the sub-feature sets created with the suggested SEM-based method in three and five-class classification provided better performance. These results showed that effective feature selection can be made with the proposed model-based approach using SEM. With this approach, it is possible to obtain sub-feature sets that form a model which statistically significant and consistent with field knowledge by including expert knowledge in the modeling process.

Keywords

• Feature selection
• Structural equation modeling
• Feature importance
• Artificial neural networks

References

1. [1] Zhao, Z. A. and Liu, H., “Spectral feature selection for data mining: 1st ed.”, Chapman and Hall/CRC, (2012). DOI: https://doi.org/10.1201/b11426
2. [2] Dong, H., Sun, J., Sun, X., Ding, R., “A many-objective feature selection for multi-label classification”, Knowledge-Based Systems, 208, (2020). DOI: https://doi.org/10.1016/j.knosys.2020.106456
3. [3] Bolón-Canedo, V., Sánchez-Maroño, N., Alonso-Betanzos, A., “Recent advances and emerging challenges of feature selection in the context of big data”, Knowledge-Based Systems, 86: 33–45, (2015). DOI: https://doi.org/10.1016/j.knosys.2015.05.014
4. [4] Li, M., Wang, H., Yang, L., Liang, Y., Shang, Z., Wan, H., “Fast hybrid dimensionality reduction method for classification based on feature selection and grouped feature extraction”, Expert Systems with Applications, 150, (2020). DOI: https://doi.org/10.1016/j.eswa.2020.113277
5. [5] Sha, Z. C., Liu, Z. M., Ma, C., Chen, J., “Feature selection for multi-label classification by maximizing full-dimensional conditional mutual information”, Applied Intelligence, 51(1): 326–340, (2021). DOI: https://doi.org/10.1007/s10489-020-01822-0
6. [6] Uysal, A. K., and Gunal, S., “A novel probabilistic feature selection method for text classification”, Knowledge-Based Systems, 36: 226–235, (2012). DOI: https://doi.org/10.1016/j.knosys.2012.06.005
7. [7] Remeseiro, B., and Bolon-Canedo, V., “A review of feature selection methods in medical applications”, Computers in Biology and Medicine, 112, (2019). DOI: https://doi.org/10.1016/j.compbiomed.2019.103375
8. [8] Jiang, Z., and Zhao, W., “Optimal selection of customized features for implementing seizure detection in wearable electroencephalography sensor”, IEEE Sensors Journal, 20(21): 12941–12949, (2020). DOI: https://doi.org/10.1109/JSEN.2020.3003733

9. [9] Bolón-Canedo, Verónica, Sánchez-Maroño, N., Alonso-Betanzos, A., “A review of feature selection methods on synthetic data”, Knowledge and Information Systems, 34(3): 483–519, (2013). DOI: https://doi.org/10.1007/s10115-012-0487-8
10. [10] Colaco, S., Kumar, S., Tamang, A., Biju, V. G., “A review on feature selection algorithms”, Advances in Intelligent Systems and Computing, Vol. 906, (2019). DOI: https://doi.org/10.1007/978-981-13-6001-5_11
11. [11] Jiliang Tang, Salem Alelyani, H. L., “Feature selection for classification: a review” (C. C. Aggarwal (ed.)). Chapman and Hall/CRC, (2014). DOI: https://doi.org/10.1201/b17320
12. [12] Monirul Kabir, M., Monirul Islam, M., Murase, K., “A new wrapper feature selection approach using neural network”, Neurocomputing, 73(16–18): 3273–3283, (2010). DOI: https://doi.org/10.1016/j.neucom.2010.04.003
13. [13] Gan, J., Wen, G., Yu, H., Zheng, W., Lei, C., “Supervised feature selection by self-paced learning regression”, Pattern Recognition Letters, 132: 30–37, (2020). DOI: https://doi.org/10.1016/j.patrec.2018.08.029
14. [14] Mazaheri, V., and Khodadadi, H., “Heart arrhythmia diagnosis based on the combination of morphological, frequency and nonlinear features of ECG signals and metaheuristic feature selection algorithm”, Expert Systems with Applications, 161, (2020). DOI: https://doi.org/10.1016/j.eswa.2020.113697
15. [15] Sahebi, G., Movahedi, P., Ebrahimi, M., Pahikkala, T., Plosila, J., Tenhunen, H., “GeFeS: A generalized wrapper feature selection approach for optimizing classification performance”, Computers in Biology and Medicine, 125, (2020). DOI: https://doi.org/10.1016/j.compbiomed.2020.103974
16. [16] Rick H. Hoyle, “Handbook of structural equation modeling”, The Guilford Press, (2014).
17. [17] Congdon, P., “Structural equation models for area health outcomes with model selection”, Journal of Applied Statistics, 38(4): 745–767, (2011). DOI: https://doi.org/10.1080/02664760903563692
18. [18] Marsh, Herbert W., Hocevar, D., “Application of confirmatory factor analysis to the study of self-concept: First- and higher order factor models and their invariance across groups”, Psychological Bulletin, 97(3): 562–582, (1985). DOI: https://doi.org/10.1037/0033-2909.97.3.562
19. [19] Bentler, P. M., “Comparative fit indexes in structural models”, Psychological Bulletin, 107(2): 238–246, (1990). DOI: https://doi.org/10.1037/0033-2909.107.2.238
20. [20] Browne, M. W., and Cudeck, R., “Alternative ways of assessing model fit”, Sociological Methods & Research, 21(2): 230–258, (1992). DOI: https://doi.org/10.1177/0049124192021002005
21. [21] Kursa, M. B., Jankowski, A., Rudnicki, W. R., “Boruta - a system for feature selection”, Fundamenta Informaticae, 101(4): 271–285, (2010). DOI: https://doi.org/10.3233/FI-2010-288
22. [22] Too, J., and Mirjalili, S., “General learning equilibrium optimizer: a new feature selection method for biological data classification”, Applied Artificial Intelligence, 35(3): 247–263, (2020). DOI: https://doi.org/10.1080/08839514.2020.1861407