Bootstrap-Driven Feature Weighting For Stable k-NN Performance

Abstract

The algorithm of k-Nearest Neighbors (k-NN) is continuous to be applied because of its simplicity, easy to understand, and its flexibility. However, it often does not work well when the dataset includes irrelevant features. This article proposes a data driven lightweight feature weighting method which uses the bootstrap sample and applies stability and predictive relevance estimates for each feature. It obtained aggregate mutual information (MI) values over resampled subsets, from which feature weights are derived and used to improve the distance metric in the k-NN, without the need for a complicated model training. The results of experiment demonstrate a significant improvement 11.19% in F1 value under noisy conditions, along with higher accuracy 1.36% (96.01% vs 94.65%) and reduced the performance variance (±5.33% vs ±7.16%) compared to the standard k-NN. The proposed method is easy to interpret and can be applied within the structure of the conventional k-NN.

Keywords

• k-nearest neighbors
• Feature weighting
• Bootstrap resampling
• Stable classification

References

1. Abdalla, H. I., & Amer, A. A. (2025). Enhancing data classification using locally informed weighted k-nearest neighbor algorithm. Expert Systems with Applications, 276, 126942. https://doi.org/10.1016/j.eswa.2025.126942
2. Ali, N., Neagu, D., & Trundle, P. (2019). Evaluation of k-nearest neighbour classifier performance for heterogeneous data sets. SN Applied Sciences, 1, 1559. https://doi.org/10.1007/s42452-019-1356-9
3. Altman, N. (1992). An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician, 46, 175–185.
4. Aurélien, G. (2019). Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow. Concepts, tools, and techniques to build intelligent systems, 2nd ednn. Jordan University of Science and Technology.
5. Biswas, N., Chakraborty, S., Mullick, S. S., & Das, S. (2018). A parameter independent fuzzy weighted k-nearest neighbor classifier. Pattern Recognition Letters, 101, 80-87. https://doi.org/10.1016/j.patrec.2017.11.003
6. Bolón-Canedo, V., Sánchez-Maroño, N., & Alonso-Betanzos, A. (2015). Recent advances and emerging challenges of feature selection in the context of big data. Knowledge-Based Systems, 86, 33-45. https://doi.org/10.1016/j.knosys.2015.05.014
7. Bommert, A., Sun, X., Bischl, B., Rahnenführer, J., & Lang, M. (2020). Benchmark for filter methods for feature selection in high-dimensional classification data. Computational Statistics & Data Analysis, 143, 106839. https://doi.org/10.1016/j.csda.2019.106839
8. Chen, Y., & Hao, Y. (2017). A feature weighted support vector machine and K-nearest neighbor algorithm for stock market indices prediction. Expert Systems with Applications, 80, 340-355. https://doi.org/10.1016/j.eswa.2017.02.044

9. Cover, T., & Hart, P. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13, 21–27. https://doi.org/10.1109/TIT.1967.1053964
10. Fisher, R. (1936). Iris [Dataset]. UCI Machine Learning Repository.
11. Gul, N., Mashwani, W. K., Aamir, M., Aldahmani, S., & Khan, Z. (2023). Optimal model selection for k-nearest neighbours ensemble via sub-bagging and sub-sampling with feature weighting. Alexandria Engineering Journal, 72, 157-168. https://doi.org/10.1016/j.aej.2023.03.075
12. Halder, R. K., Uddin, M. N., Uddin, M. A., Aryal, S., & Khraisat, A. (2024). Enhancing K-nearest neighbor algorithm: a comprehensive review and performance analysis of modifications. Journal of Big Data, 11(1), 113. https://doi.org/10.1186/s40537-024-00973-y
13. Ircio, J., Lojo, A., Mori, U., & Lozano, J. A. (2020). Mutual information based feature subset selection in multivariate time series classification. Pattern Recognition, 108, 107525.
14. Łukaszuk, T., Krawczuk, J., Żyła, K., & Kęsik, J. (2024). Stability of feature selection in multi-omics data analysis. Applied Sciences, 14, 11103. https://doi.org/10.3390/app142311103
15. Peng, H., Long, F., & Ding, C. (2005). Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8), 1226-1238. https://doi.org/10.1109/TPAMI.2005.159
16. Popoola, G., Fuhnwi, G., Agbaje, J., & Fesomade, K. (2024). A modified ant colony optimization with KNN for high-dimensional data classification. In: Arai, K. (eds) Intelligent Computing. SAI 2024. Lecture Notes in Networks and Systems, vol 1018. Springer, Cham. https://doi.org/10.1007/978-3-031-62269-4_19
17. Pudjihartono, N., Fadason, T., Kempa-Liehr, A. W., & O'Sullivan, J. M. (2022). A review of feature selection methods for machine learning-based disease risk prediction. Frontiers in Bioinformatics, 2, 927312. https://doi.org/10.3389/fbinf.2022.927312
18. Saeys, Y., Inza, I., & Larranaga, P. (2007). A review of feature selection techniques in bioinformatics. Bioinformatics, 23(19), 2507-2517. https://doi.org/10.1093/bioinformatics/btm344
19. Salman, R., Alzaatreh, A., Sulieman, H., & Faisal, S. (2021). A bootstrap framework for aggregating within and between feature selection methods. Entropy, 23, 200. https://doi.org/10.3390/e23020200
20. Uddin, S., Haque, I., Lu, H., Moni, M. A., & Gide, E. (2022). Comparative performance analysis of K-nearest neighbour (KNN) algorithm and its different variants for disease prediction. Scientific Reports, 12(1), 6256. https://doi.org/10.1038/s41598-022-10358-x
21. Venkateswara, H., Lade, P., Lin, B., Ye, J., & Panchanathan, S. (2015). Efficient approximate solutions to mutual information based global feature selection. IEEE International Conference on Data Mining (pp. 1009–1014). IEEE. https://doi.org/10.1109/ICDM.2015.140
22. Vergara, J. R., & Estévez, P. A. (2014). A review of feature selection methods based on mutual information. Neural Computing and Applications, 24, 175–186. https://doi.org/10.1007/s00521-013-1368-0