A Comprehensive Review of Feature Selection and Feature Selection Stability in Machine Learning

Abstract

Feature selection is a dimension reduction technique used to select features that are relevant to machine learning tasks. Reducing the dataset size by eliminating redundant and irrelevant features plays a pivotal role in increasing the performance of machine learning algorithms, speeding up the learning process, and building simple models. The apparent need for feature selection has aroused considerable interest amongst researchers and has caused feature selection to find a wide range of application domains including text mining, pattern recognition, cybersecurity, bioinformatics, and big data. As a result, over the years, a substantial amount of literature has been published on feature selection and a wide variety of feature selection methods have been proposed. The quality of feature selection algorithms is measured not only by evaluating the quality of the models built using the features they select, or by the clustering tendencies of the features they select, but also by their stability. Therefore, this study focused on feature selection and feature selection stability. In the pages that follow, general concepts and methods of feature selection, feature selection stability, stability measures, and reasons and solutions for instability are discussed.

Keywords

• Feature selection
• Dimensionality reduction
• Types of feature selection
• Feature selection stability
• Stability measures

References

1. [1] Kohavi, R., John, G.H., “Wrappers for feature subset selection”, Artificial Intelligence, 97(1-2): 273-324, (1997).
2. [1] Kohavi, R., John, G.H., “Wrappers for feature subset selection”, Artificial Intelligence, 97(1-2): 273-324, (1997).
3. [2] Yu, L., Liu, H., “Efficient Feature Selection via Analysis of Relevance and Redundancy”, Journal of Machine Learning Research, 5: 1205-1224, (2004).
4. [2] Yu, L., Liu, H., “Efficient Feature Selection via Analysis of Relevance and Redundancy”, Journal of Machine Learning Research, 5: 1205-1224, (2004).
5. [3] Yu, L., Liu, H., “Redundancy Based Feature Selection for Microarray Data”, KDD ‘04: Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, WA, USA, 737-742, (2004).
6. [3] Yu, L., Liu, H., “Redundancy Based Feature Selection for Microarray Data”, KDD ‘04: Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, WA, USA, 737-742, (2004).
7. [4] Cho, S.-B., Won, H.-H., “Machine Learning in DNA Microarray Analysis for Cancer Classification”, APBC ‘03: Proceedings of the First Asia-Pacific Bioinformatics Conference on Bioinformatics, Adelaide, SA, Australia, 19: 189-198, (2003).
8. [4] Cho, S.-B., Won, H.-H., “Machine Learning in DNA Microarray Analysis for Cancer Classification”, APBC ‘03: Proceedings of the First Asia-Pacific Bioinformatics Conference on Bioinformatics, Adelaide, SA, Australia, 19: 189-198, (2003).

9. [5] Tang, J., Zhou, S., “A new approach for feature selection from microarray data based on mutual information”, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 13(6): 1004-1015, (2016).
10. [5] Tang, J., Zhou, S., “A new approach for feature selection from microarray data based on mutual information”, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 13(6): 1004-1015, (2016).
11. [6] Inza, I., Larranaga, P., Blanco, R., Cerrolaza, A.J., “Filter versus wrapper gene selection approaches in DNA microarray domains”, Artificial Intelligence in Medicine, 31(2): 91-103, (2004).
12. [6] Inza, I., Larranaga, P., Blanco, R., Cerrolaza, A.J., “Filter versus wrapper gene selection approaches in DNA microarray domains”, Artificial Intelligence in Medicine, 31(2): 91-103, (2004).
13. [7] Yang, Q., Jia, X., Li, X., Feng, J., Li, W., Lee, J., “Evaluating feature selection and anomaly detection methods of hard drive failure prediction”, IEEE Transactions on Reliability, 70(2): 749-760, (2021).
14. [7] Yang, Q., Jia, X., Li, X., Feng, J., Li, W., Lee, J., “Evaluating feature selection and anomaly detection methods of hard drive failure prediction”, IEEE Transactions on Reliability, 70(2): 749-760, (2021).
15. [8] Lee, W., Stolfo, S.J., Mok, K.W., “Adaptive intrusion detection: a data mining approach”, Artificial Intelligence Review, 14: 533-567, (2000).
16. [8] Lee, W., Stolfo, S.J., Mok, K.W., “Adaptive intrusion detection: a data mining approach”, Artificial Intelligence Review, 14: 533-567, (2000).
17. [9] Alazab, A., Hobbs, M., Abawajy, J., Alazab, M., “Using Feature Selection for Intrusion Detection System”, International Symposium on Communications and Information Technologies (ISCIT), Gold Coast, QLD, Australia, 296-301, (2012).
18. [9] Alazab, A., Hobbs, M., Abawajy, J., Alazab, M., “Using Feature Selection for Intrusion Detection System”, International Symposium on Communications and Information Technologies (ISCIT), Gold Coast, QLD, Australia, 296-301, (2012).
19. [10] Huang, K., Aviyente, S., “Wavelet feature selection for image classification”, IEEE Transactions on Image Processing, 17(9): 1709-1720, (2008).
20. [10] Huang, K., Aviyente, S., “Wavelet feature selection for image classification”, IEEE Transactions on Image Processing, 17(9): 1709-1720, (2008).
21. [11] Dy, J.G., Brodley, C.E., Kak, A., Broderick, L.S., Aisen, A.M., “Unsupervised feature selection applied to content-based retrieval of lung images”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(3): 373-378, (2003).
22. [11] Dy, J.G., Brodley, C.E., Kak, A., Broderick, L.S., Aisen, A.M., “Unsupervised feature selection applied to content-based retrieval of lung images”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(3): 373-378, (2003).
23. [12] Forman, G., “An Extensive Empirical Study of Feature Selection Metrics for Text Classification”, Journal of Machine Learning Research, 3: 1289-1305, (2003).
24. [12] Forman, G., “An Extensive Empirical Study of Feature Selection Metrics for Text Classification”, Journal of Machine Learning Research, 3: 1289-1305, (2003).
25. [13] Jing, L.-P., Huang, H.-K., Shi, H.-B., “Improved Feature Selection Approach TFIDF in Text Mining”, Proceedings of the International Conference on Machine Learning and Cybernetics, Beijing, China, 944-946, (2002).
26. [13] Jing, L.-P., Huang, H.-K., Shi, H.-B., “Improved Feature Selection Approach TFIDF in Text Mining”, Proceedings of the International Conference on Machine Learning and Cybernetics, Beijing, China, 944-946, (2002).
27. [14] Bai, X., Gao, X., Xue, B., “Particle swarm optimization based two-stage feature selection in text mining”, 2018 IEEE Congress on Evolutionary Computation (CEC), 1-8, (2018).
28. [14] Bai, X., Gao, X., Xue, B., “Particle swarm optimization based two-stage feature selection in text mining”, 2018 IEEE Congress on Evolutionary Computation (CEC), 1-8, (2018).
29. [15] Fisher, R.A., “The use of multiple measurements in taxonomic problems”, Annals of Eugenics, 7: 179-188, (1936).
30. [15] Fisher, R.A., “The use of multiple measurements in taxonomic problems”, Annals of Eugenics, 7: 179-188, (1936).
31. [16] Han, D., Kim, J., “Unified simultaneous clustering and feature selection for unlabeled and labeled data”, IEEE Transactions on Neural Networks and Learning Systems, 29(12): 6083-6098, (2018).
32. [16] Han, D., Kim, J., “Unified simultaneous clustering and feature selection for unlabeled and labeled data”, IEEE Transactions on Neural Networks and Learning Systems, 29(12): 6083-6098, (2018).
33. [17] Zhao, Z., Liu, H., “Spectral Feature Selection for Supervised and Unsupervised Learning”, ICML ‘07: Proceedings of the 24th International Conference on Machine Learning, Corvalis, OR, USA, 1151-1157, (2007).
34. [17] Zhao, Z., Liu, H., “Spectral Feature Selection for Supervised and Unsupervised Learning”, ICML ‘07: Proceedings of the 24th International Conference on Machine Learning, Corvalis, OR, USA, 1151-1157, (2007).
35. [18] Tang, J., Alelyani, S., Liu, H., “Feature selection for classification: a review”, Data Classification: Algorithms and Applications, CRC Press, 37-64, (2014).
36. [18] Tang, J., Alelyani, S., Liu, H., “Feature selection for classification: a review”, Data Classification: Algorithms and Applications, CRC Press, 37-64, (2014).
37. [19] Ang, J.C., Mirzal, A., Haron, H., Hamed, H.N.A., “Supervised, unsupervised, and semi-supervised feature selection: a review on gene selection”, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 13(5): 971-989, (2015).
38. [19] Ang, J.C., Mirzal, A., Haron, H., Hamed, H.N.A., “Supervised, unsupervised, and semi-supervised feature selection: a review on gene selection”, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 13(5): 971-989, (2015).
39. [20] Yang, W., Wang, K., Zuo, W., “Neighborhood Component Feature Selection for High-Dimensional Data”, Journal of Computers, 7(1): 161-168, (2012).
40. [20] Yang, W., Wang, K., Zuo, W., “Neighborhood Component Feature Selection for High-Dimensional Data”, Journal of Computers, 7(1): 161-168, (2012).
41. [21] Dy, J.G., Brodley, C.E., Wrobel, S. (Editor), “Feature Selection for Unsupervised Learning”, The Journal of Machine Learning Research, 5: 845-889, (2004).
42. [21] Dy, J.G., Brodley, C.E., Wrobel, S. (Editor), “Feature Selection for Unsupervised Learning”, The Journal of Machine Learning Research, 5: 845-889, (2004).
43. [22] Solorio-Fernandez, S., Carrasco-Ochoa, J.A., Martinez-Trinidad, J.F., “A review of unsupervised feature selection methods”, Artificial Intelligence Review, 53: 907-948, (2020).
44. [22] Solorio-Fernandez, S., Carrasco-Ochoa, J.A., Martinez-Trinidad, J.F., “A review of unsupervised feature selection methods”, Artificial Intelligence Review, 53: 907-948, (2020).
45. [23] Boutsidis, C., Mahoney, M.W., Drineas, P., “Unsupervised Feature Selection for Principal Components Analysis”, Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, NV, USA, 61-69, (2008).
46. [23] Boutsidis, C., Mahoney, M.W., Drineas, P., “Unsupervised Feature Selection for Principal Components Analysis”, Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, NV, USA, 61-69, (2008).
47. [24] He, X., Cai, D., Niyogi, P., “Laplacian Score for Feature Selection”, NIPS ‘05: Proceedings of the 18th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 507-514, (2005).
48. [24] He, X., Cai, D., Niyogi, P., “Laplacian Score for Feature Selection”, NIPS ‘05: Proceedings of the 18th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 507-514, (2005).
49. [25] Zhao, Z., Liu, H., “Semi-supervised Feature Selection via Spectral Analysis”, Proceedings of the 7th SIAM International Conference on Data Mining, Minneapolis, MN, USA, 641-646, (2007).
50. [25] Zhao, Z., Liu, H., “Semi-supervised Feature Selection via Spectral Analysis”, Proceedings of the 7th SIAM International Conference on Data Mining, Minneapolis, MN, USA, 641-646, (2007).
51. [26] Ren, J., Qiu, Z., Fan, W., Cheng, H., Yu, P.S., “Forward semi-supervised feature selection”, PAKDD ‘08: Advances in Knowledge Discovery and Data Mining, Lecture Notes in Computer Science, 5012: 970-976, (2008).
52. [26] Ren, J., Qiu, Z., Fan, W., Cheng, H., Yu, P.S., “Forward semi-supervised feature selection”, PAKDD ‘08: Advances in Knowledge Discovery and Data Mining, Lecture Notes in Computer Science, 5012: 970-976, (2008).
53. [27] Sheikhpour, R., Sarram, M.A., Gharaghani, S., Chahooki, M.A.Z., “A Survey on semi-supervised feature selection methods”, Pattern Recognition, 64: 141-158, (2017).
54. [27] Sheikhpour, R., Sarram, M.A., Gharaghani, S., Chahooki, M.A.Z., “A Survey on semi-supervised feature selection methods”, Pattern Recognition, 64: 141-158, (2017).
55. [28] Xu, Z., King, I., Lyu, M.R., Jin, R., “Discriminative semi-supervised feature selection via manifold regularization”, IEEE Transactions on Neural Networks, 21(7): 1303-1308, (2010).
56. [28] Xu, Z., King, I., Lyu, M.R., Jin, R., “Discriminative semi-supervised feature selection via manifold regularization”, IEEE Transactions on Neural Networks, 21(7): 1303-1308, (2010).
57. [29] Zhao, J., Lu, K., He, X., “Locality sensitive semi-supervised feature selection”, Neurocomputing, 71(10-12): 1842-1849, (2008).
58. [29] Zhao, J., Lu, K., He, X., “Locality sensitive semi-supervised feature selection”, Neurocomputing, 71(10-12): 1842-1849, (2008).
59. [30] Guyon, I., Elisseeff, A., Kaelbling, L.P. (Editor), “An Introduction to Variable and Feature Selection”, Journal of Machine Learning Research, 3: 1157-1182, (2003).
60. [30] Guyon, I., Elisseeff, A., Kaelbling, L.P. (Editor), “An Introduction to Variable and Feature Selection”, Journal of Machine Learning Research, 3: 1157-1182, (2003).
61. [31] Haury, A.-C., Gestraud, P., Vert, J.-P., “The influence of feature selection methods on accuracy, stability and interpretability of molecular signatures”, PLoS ONE, 6(12): e28210, (2011).
62. [31] Haury, A.-C., Gestraud, P., Vert, J.-P., “The influence of feature selection methods on accuracy, stability and interpretability of molecular signatures”, PLoS ONE, 6(12): e28210, (2011).
63. [32] Breiman, L., Friedman, J.H., Stone, C.J., Olshen, R.A., “Classification and regression trees”, 1st Ed., United Kingdom: Chapman and Hall/CRC, 18-55, 216-264, (1984).
64. [32] Breiman, L., Friedman, J.H., Stone, C.J., Olshen, R.A., “Classification and regression trees”, 1st Ed., United Kingdom: Chapman and Hall/CRC, 18-55, 216-264, (1984).
65. [33] Quinlan, J.R., “Induction of decision trees”, Machine Learning, 1: 81-106, (1986).
66. [33] Quinlan, J.R., “Induction of decision trees”, Machine Learning, 1: 81-106, (1986).
67. [34] Tharwat, A., “Classification assessment methods: a detailed tutorial”, Applied Computing and Informatics, (2018).
68. [34] Tharwat, A., “Classification assessment methods: a detailed tutorial”, Applied Computing and Informatics, (2018).
69. [35] Landgrebe, T.C.W., Duin, R.P.W., “Approximating the multiclass ROC by pairwise analysis”, Pattern Recognition Letters, 28(13): 1747-1758, (2007).
70. [35] Landgrebe, T.C.W., Duin, R.P.W., “Approximating the multiclass ROC by pairwise analysis”, Pattern Recognition Letters, 28(13): 1747-1758, (2007).
71. [36] Fawcett, T., “An introduction to ROC analysis”, Pattern Recognition Letters, 27(8): 861-874, (2006).
72. [36] Fawcett, T., “An introduction to ROC analysis”, Pattern Recognition Letters, 27(8): 861-874, (2006).
73. [37] Turney, P., “Technical note: bias and the quantification of stability”, Machine Learning, 20, 23-33, (1995).
74. [37] Turney, P., “Technical note: bias and the quantification of stability”, Machine Learning, 20, 23-33, (1995).
75. [38] Hulse, J.V., Khoshgoftaar, T.M., Napolitano, A., Wald, R., “Feature Selection with High-Dimensional Imbalanced Data”, 2009 IEEE International Conference on Data Mining Workshops, Miami, FL, USA, 507-514, (2009).
76. [38] Hulse, J.V., Khoshgoftaar, T.M., Napolitano, A., Wald, R., “Feature Selection with High-Dimensional Imbalanced Data”, 2009 IEEE International Conference on Data Mining Workshops, Miami, FL, USA, 507-514, (2009).
77. [39] Maldonado, S., Weber, R., Famili, F., “Feature selection for high-dimensional class-imbalanced data sets using support vector machines”, Information Sciences, 286: 228-246, (2014).
78. [39] Maldonado, S., Weber, R., Famili, F., “Feature selection for high-dimensional class-imbalanced data sets using support vector machines”, Information Sciences, 286: 228-246, (2014).
79. [40] Viegas, F., Rocha, L., Gonçalves, M., Mourao, F., Sa, G., Salles, T., Andrade, G., Sandin, I., “A genetic programming approach for feature selection in highly dimensional skewed data”, Neurocomputing, 273: 554-569, (2018).
80. [40] Viegas, F., Rocha, L., Gonçalves, M., Mourao, F., Sa, G., Salles, T., Andrade, G., Sandin, I., “A genetic programming approach for feature selection in highly dimensional skewed data”, Neurocomputing, 273: 554-569, (2018).
81. [41] Katrutsa, A., Strijov, V., “Comprehensive study of feature selection methods to solve multicollinearity problem according to evaluation criteria”, Expert Systems with Applications, 76: 1-15, (2017).
82. [41] Katrutsa, A., Strijov, V., “Comprehensive study of feature selection methods to solve multicollinearity problem according to evaluation criteria”, Expert Systems with Applications, 76: 1-15, (2017).
83. [42] Jain, A., Zongker, D., “Feature selection: evaluation, application, and small sample performance”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(2): 153-158, (1997).
84. [42] Jain, A., Zongker, D., “Feature selection: evaluation, application, and small sample performance”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(2): 153-158, (1997).
85. [43] Wu, X., Cheng, Q., “Algorithmic Stability and Generalization of an Unsupervised FSA”, NeurIPS 2021: 35th Conference on Neural Information Processing Systems, 1-14, (2021).
86. [43] Wu, X., Cheng, Q., “Algorithmic Stability and Generalization of an Unsupervised FSA”, NeurIPS 2021: 35th Conference on Neural Information Processing Systems, 1-14, (2021).
87. [44] Helleputte, T., Dupont, P., “Partially Supervised Feature Selection with Regularized Linear Models”, ICML ‘09: Proceedings of the 26th Annual International Conference on Machine Learning, 409-416, (2009).
88. [44] Helleputte, T., Dupont, P., “Partially Supervised Feature Selection with Regularized Linear Models”, ICML ‘09: Proceedings of the 26th Annual International Conference on Machine Learning, 409-416, (2009).
89. [45] Lai, D.T.C., Garibaldi, J.M., “Improving Semi-supervised Fuzzy C-Means Classification of Breast Cancer Data Using Feature Selection”, 2013 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Hyderabad, India, 1-8, (2013).
90. [45] Lai, D.T.C., Garibaldi, J.M., “Improving Semi-supervised Fuzzy C-Means Classification of Breast Cancer Data Using Feature Selection”, 2013 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Hyderabad, India, 1-8, (2013).
91. [46] Kalousis, A., Prados, J., Hilario, M., “Stability of feature selection algorithms: a study on high-dimensional spaces”, Knowledge and Information Systems, 12: 95-116, (2007).
92. [46] Kalousis, A., Prados, J., Hilario, M., “Stability of feature selection algorithms: a study on high-dimensional spaces”, Knowledge and Information Systems, 12: 95-116, (2007).
93. [47] Ding, C., Peng, H., “Minimum Redundancy Feature Selection from Microarray Gene Expression Data”, Journal of Bioinformatics and Computational Biology, 3(2): 185-205, (2005).
94. [47] Ding, C., Peng, H., “Minimum Redundancy Feature Selection from Microarray Gene Expression Data”, Journal of Bioinformatics and Computational Biology, 3(2): 185-205, (2005).
95. [48] Shabbir, A., Javed, K., Ansari, Y., Babri, H.A., “Stability of Feature Ranking Algorithms on Binary Data”, Pakistan Journal of Engineering and Applied Sciences, 15: 76-86, (2014).
96. [48] Shabbir, A., Javed, K., Ansari, Y., Babri, H.A., “Stability of Feature Ranking Algorithms on Binary Data”, Pakistan Journal of Engineering and Applied Sciences, 15: 76-86, (2014).
97. [49] Jurman, G., Merler, S., Barla, A., Paoli, S., Galea, A., Furlanello, C., “Algebraic stability indicators for ranked lists in molecular profiling”, Bioinformatics, 24(2): 258-264, (2008).
98. [49] Jurman, G., Merler, S., Barla, A., Paoli, S., Galea, A., Furlanello, C., “Algebraic stability indicators for ranked lists in molecular profiling”, Bioinformatics, 24(2): 258-264, (2008).
99. [50] Kononenko, I., Simec, E., Robnik-Sikonja, M., “Overcoming the myopia of inductive learning algorithms with RELIEFF”, Applied Intelligence, 7: 39-55, (1997).
100. [50] Kononenko, I., Simec, E., Robnik-Sikonja, M., “Overcoming the myopia of inductive learning algorithms with RELIEFF”, Applied Intelligence, 7: 39-55, (1997).
101. [51] Saeys, Y., Abeel T., Van de Peer, Y., “Robust feature selection using ensemble feature selection techniques”, ECML PKDD ‘08: Machine Learning and Knowledge Discovery in Databases, 5212: 313-325, (2008).
102. [51] Saeys, Y., Abeel T., Van de Peer, Y., “Robust feature selection using ensemble feature selection techniques”, ECML PKDD ‘08: Machine Learning and Knowledge Discovery in Databases, 5212: 313-325, (2008).
103. [52] Yu, L., Ding, C., Loscalzo, S., “Stable Feature Selection via Dense Feature Groups”, KDD ‘08: Proceeding of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, NV, USA, 803-811, (2008).
104. [52] Yu, L., Ding, C., Loscalzo, S., “Stable Feature Selection via Dense Feature Groups”, KDD ‘08: Proceeding of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, NV, USA, 803-811, (2008).
105. [53] Kuncheva, L.I., “A Stability Index for Feature Selection”, Proceedings of the 25th IASTED International Conference on Artificial Intelligence and Applications, Innsbruck, Austria, 390-395, (2007).
106. [53] Kuncheva, L.I., “A Stability Index for Feature Selection”, Proceedings of the 25th IASTED International Conference on Artificial Intelligence and Applications, Innsbruck, Austria, 390-395, (2007).
107. [54] Dunne, K., Cunningham, P., Azuaje, F., “Solutions to Instability Problems with Sequential Wrapper-based Approaches to Feature Selection”, Journal of Machine Learning Research, 1-22, (2002).
108. [54] Dunne, K., Cunningham, P., Azuaje, F., “Solutions to Instability Problems with Sequential Wrapper-based Approaches to Feature Selection”, Journal of Machine Learning Research, 1-22, (2002).
109. [55] Lustgarten, J.L., Gopalakrishnan, V., Visweswaran, S., “Measuring Stability of Feature Selection in Biomedical Datasets”, AMIA ‘09: Annual Symposium Proceedings, Published Online, 406-410, (2009).
110. [55] Lustgarten, J.L., Gopalakrishnan, V., Visweswaran, S., “Measuring Stability of Feature Selection in Biomedical Datasets”, AMIA ‘09: Annual Symposium Proceedings, Published Online, 406-410, (2009).
111. [56] Zucknick, M., Richardson, S., Stronach, E.A., “Comparing the characteristics of gene expression profiles derived by univariate and multivariate classification methods”, Statistical Applications in Genetics and Molecular Biology, 7(1): 1-28, (2008).
112. [56] Zucknick, M., Richardson, S., Stronach, E.A., “Comparing the characteristics of gene expression profiles derived by univariate and multivariate classification methods”, Statistical Applications in Genetics and Molecular Biology, 7(1): 1-28, (2008).
113. [57] Shi, L., Tong, W., Fang, H., Scherf, U., Han, J., Puri, R.K., Frueh, F.W., Goodsaid, F.M., Guo, L., Su, Z., Han, T., Fuscoe, J.C., Xu, Z.A., Patterson, T.A., Hong, H., Xie, Q., Perkins, R.G., Chen, J.J., Casciano, D.A., “Cross-platform comparability of microarray technology: intraplatform consistency and appropriate data analysis procedures are essential”, BMC Bioinformatics 6, Article number S12, (2005).
114. [57] Shi, L., Tong, W., Fang, H., Scherf, U., Han, J., Puri, R.K., Frueh, F.W., Goodsaid, F.M., Guo, L., Su, Z., Han, T., Fuscoe, J.C., Xu, Z.A., Patterson, T.A., Hong, H., Xie, Q., Perkins, R.G., Chen, J.J., Casciano, D.A., “Cross-platform comparability of microarray technology: intraplatform consistency and appropriate data analysis procedures are essential”, BMC Bioinformatics 6, Article number S12, (2005).
115. [58] Zhang, M., Zhang, L., Zou, J., Yao, C., Xiao, H., Liu, Q., Wang, J., Wang, D., Wang, C., Guo, Z., “Evaluating reproducibility of differential expression discoveries in microarray studies by considering correlated molecular changes”, Bioinformatics, 25(13): 1662-1668, (2009).
116. [58] Zhang, M., Zhang, L., Zou, J., Yao, C., Xiao, H., Liu, Q., Wang, J., Wang, D., Wang, C., Guo, Z., “Evaluating reproducibility of differential expression discoveries in microarray studies by considering correlated molecular changes”, Bioinformatics, 25(13): 1662-1668, (2009).
117. [59] Wald, R., Khoshgoftaar, T., Dittman, D., “A New Fixed-overlap Partitioning Algorithm for Determining Stability of Bioinformatics Gene Rankers”, 11th International Conference on Machine Learning and Applications (ICMLA), Boca Raton, FL, USA, 170-177, (2012).
118. [59] Wald, R., Khoshgoftaar, T., Dittman, D., “A New Fixed-overlap Partitioning Algorithm for Determining Stability of Bioinformatics Gene Rankers”, 11th International Conference on Machine Learning and Applications (ICMLA), Boca Raton, FL, USA, 170-177, (2012).
119. [60] Gulgezen, G., Cataltepe, Z., Yu., L., “Stable and accurate feature selection”, ECML PKDD ‘09: Machine Learning and Knowledge Discovery in Databases, 5781: 455-468, (2009).
120. [60] Gulgezen, G., Cataltepe, Z., Yu., L., “Stable and accurate feature selection”, ECML PKDD ‘09: Machine Learning and Knowledge Discovery in Databases, 5781: 455-468, (2009).
121. [61] Nogueira, S., “Quantifying the stability of feature selection”, Ph.D. Thesis, University of Manchester, Manchester, United Kingdom, 21-67, (2018).
122. [61] Nogueira, S., “Quantifying the stability of feature selection”, Ph.D. Thesis, University of Manchester, Manchester, United Kingdom, 21-67, (2018).
123. [62] Lausser, L., Müssel, C., Maucher, M., Kestler, H.A., “Measuring and visualizing the stability of biomarker selection techniques”, Computational Statistics, 28: 51-65, (2013).
124. [62] Lausser, L., Müssel, C., Maucher, M., Kestler, H.A., “Measuring and visualizing the stability of biomarker selection techniques”, Computational Statistics, 28: 51-65, (2013).
125. [63] Krizek, P., Kittler, J., Hlavac, V., “Improving Stability of Feature Selection Methods”, 12th International Conference on Computer Analysis of Images and Patterns (CAIP), Vienna, Austria, 929-936, (2007).
126. [63] Krizek, P., Kittler, J., Hlavac, V., “Improving Stability of Feature Selection Methods”, 12th International Conference on Computer Analysis of Images and Patterns (CAIP), Vienna, Austria, 929-936, (2007).
127. [64] Guzman-Martinez, R., Alaiz-Rodriguez, R., “Feature selection stability assessment based on the Jensen-Shannon divergence”, Lecture Notes in Computer Science, 6911: 597-612, (2011).
128. [64] Guzman-Martinez, R., Alaiz-Rodriguez, R., “Feature selection stability assessment based on the Jensen-Shannon divergence”, Lecture Notes in Computer Science, 6911: 597-612, (2011).
129. [65] Davis, C.A., Gerick, F., Hintermair, V., Friedel, C.C., Fundel, K., Küffner, R., Zimmer, R., “Reliable gene signatures for microarray classification: assessment of stability and performance”, Bioinformatics, 22(19): 2356-2363, (2006).
130. [65] Davis, C.A., Gerick, F., Hintermair, V., Friedel, C.C., Fundel, K., Küffner, R., Zimmer, R., “Reliable gene signatures for microarray classification: assessment of stability and performance”, Bioinformatics, 22(19): 2356-2363, (2006).
131. [66] Goh, W.W.B., Wong, L., “Evaluating Feature Selection Stability in Next-Generation Proteomics”, Journal of Bioinformatics and Computational Biology, 14(5): 1650029, (2016).
132. [66] Goh, W.W.B., Wong, L., “Evaluating Feature Selection Stability in Next-Generation Proteomics”, Journal of Bioinformatics and Computational Biology, 14(5): 1650029, (2016).
133. [67] Nogueira, S., Brown, G., “Measuring the stability of feature selection”, ECML PKDD ‘16: Machine Learning and Knowledge Discovery in Databases, 9852: 442-457, (2016).
134. [67] Nogueira, S., Brown, G., “Measuring the stability of feature selection”, ECML PKDD ‘16: Machine Learning and Knowledge Discovery in Databases, 9852: 442-457, (2016).
135. [68] Munson, M.A., Caruana, R., “On feature selection, bias-variance, and bagging”, ECML PKDD ‘09: Machine Learning and Knowledge Discovery in Databases, 5782: 144-159, (2009).
136. [68] Munson, M.A., Caruana, R., “On feature selection, bias-variance, and bagging”, ECML PKDD ‘09: Machine Learning and Knowledge Discovery in Databases, 5782: 144-159, (2009).
137. [69] Alelyani, S., “On feature selection stability: a data perspective”, Ph.D. Thesis, Arizona State University, Phoenix, USA, 10-57, (2013).
138. [69] Alelyani, S., “On feature selection stability: a data perspective”, Ph.D. Thesis, Arizona State University, Phoenix, USA, 10-57, (2013).
139. [70] Alelyani, S., Liu, H., Wang, L., “The Effect of the Characteristics of the Dataset on the Selection Stability”, 2011 IEEE 23rd International Conference on Tools with Artificial Intelligence, Boca Raton, FL, USA, 970-977, (2011).
140. [70] Alelyani, S., Liu, H., Wang, L., “The Effect of the Characteristics of the Dataset on the Selection Stability”, 2011 IEEE 23rd International Conference on Tools with Artificial Intelligence, Boca Raton, FL, USA, 970-977, (2011).
141. [71] Dittman, D., Khoshgoftaar, T., Wald, R., Napolitano, A., “Similarity Analysis of Feature Ranking Techniques on Imbalanced DNA Microarray Datasets”, 2012 IEEE International Conference on Bioinformatics and Biomedicine, Philadelphia, PA, USA, 1-5, (2012).
142. [71] Dittman, D., Khoshgoftaar, T., Wald, R., Napolitano, A., “Similarity Analysis of Feature Ranking Techniques on Imbalanced DNA Microarray Datasets”, 2012 IEEE International Conference on Bioinformatics and Biomedicine, Philadelphia, PA, USA, 1-5, (2012).
143. [72] Alelyani, S., Zhao, Z., Liu, H., “A Dilemma in Assessing Stability of Feature Selection Algorithms”, 2011 IEEE International Conference on High Performance Computing and Communications, Banff, AB, Canada, 701-707, (2011).
144. [72] Alelyani, S., Zhao, Z., Liu, H., “A Dilemma in Assessing Stability of Feature Selection Algorithms”, 2011 IEEE International Conference on High Performance Computing and Communications, Banff, AB, Canada, 701-707, (2011).
145. [73] Han, Y., Yu, L., “A Variance Reduction Framework for Stable Feature Selection”, 2010 IEEE International Conference on Data Mining, Sydney, NSW, Australia, 206-215, (2010).
146. [73] Han, Y., Yu, L., “A Variance Reduction Framework for Stable Feature Selection”, 2010 IEEE International Conference on Data Mining, Sydney, NSW, Australia, 206-215, (2010).
147. [74] Kamkar, I., “Building stable predictive models for healthcare applications: a data-driven approach”, Ph.D. Thesis, Deakin University, Geelong, Australia, 34-52, (2016).
148. [74] Kamkar, I., “Building stable predictive models for healthcare applications: a data-driven approach”, Ph.D. Thesis, Deakin University, Geelong, Australia, 34-52, (2016).
149. [75] Tang, F., Adam, L., Si, B., “Group feature selection with multiclass support vector machine”, Neurocomputing, 317: 42-49, (2018).
150. [75] Tang, F., Adam, L., Si, B., “Group feature selection with multiclass support vector machine”, Neurocomputing, 317: 42-49, (2018).
151. [76] Loscalzo, S., Yu, L., Ding, C.H.Q., “Consensus Group Stable Feature Selection”, Conference: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France, 567-575, (2009).
152. [76] Loscalzo, S., Yu, L., Ding, C.H.Q., “Consensus Group Stable Feature Selection”, Conference: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France, 567-575, (2009).