Simultaneous Feature Selection and Hyperparameter Tuning of K-Nearest Neighbors for Glass Classification: A Comparative Study of Artificial Bee Colony, Simulated Annealing, and Hill Climbing

Abstract

Glass identification is essential in forensic science and industrial uses. However, the effectiveness of classification algorithms heavily relies on selecting the correct hyperparameters and relevant features. This study investigates the efficacy of three optimization algorithms—Artificial Bee Colony (ABC), Simulated Annealing (SA), and Hill Climbing (HC)—for optimizing the k value, which is the neighbor value of the k-Nearest Neighbors (k-NN) classifier, the distance measure expressing the proximity between two samples, and the feature subset of the UCI Glass Identification dataset. By reframing the classification problem as a multi-dimensional optimization task, the algorithms are assessed based on accuracy, precision, recall, and F1 score. The results show that the population-based ABC algorithm, as a meta-heuristic approach, outperforms local search methods by avoiding local optima and effectively reducing the dimensionality of the data. RI (refractive index) and Mg (magnesium oxide content) emerge as particularly discriminative features. The study demonstrates that combining automatic parameter tuning with feature selection can substantially enhance the predictive performance of k-NN in complex classification tasks.

Keywords

• Glass classification
• K-Nearest neighbors
• Artificial bee colony
• Simulated annealing
• Hill climbing
• Optimization

References

1. Al-Betar, M. A., Hammouri, A. I., Awadallah, M. A., & Doush, I. A. (2021). Binary β-hill climbing optimizer with S-shape transfer function for feature selection. Journal of Ambient Intelligence and Humanized Computing, 12(7), 7637–7665. https://doi.org/10.1007/s12652-020-02484-z
2. Ay, Ş., Ekinci, E., & Garip, Z. (2023). A comparative analysis of meta-heuristic optimization algorithms for feature selection on ML-based classification of heart-related diseases. The Journal of Supercomputing, 79(11), 11797–11826. https://doi.org/10.1007/s11227-023-05132-3
3. Bai, X., Zheng, Y., Lu, Y., & Shi, Y. (2024). Chain hybrid feature selection algorithm based on improved Grey Wolf Optimization algorithm. PLOS ONE, 19(10), e0311602. https://doi.org/10.1371/journal.pone.0311602
4. Bhowmick, S., & Saha, A. (2023). Enhancing the performance of kNN for glass identification dataset using inverse distance weight, ReliefF ranking, and SMOTE. AIP Conference Proceedings, 2754(1), 020021. https://doi.org/10.1063/5.0161083
5. Bouziane, M., Bouziane, A., Larguech, S., Naima, K., Haque, M. S., & Menni, Y. (2025). High-performance glass classification using advanced machine learning and deep learning algorithms with a comprehensive feature analysis. AIP Advances, 15(5), 055013. https://doi.org/10.1063/5.0260868
6. Chantar, H., Tubishat, M., Essgaer, M., & Mirjalili, S. (2021). Hybrid binary dragonfly algorithm with simulated annealing for feature selection. SN Computer Science, 2(4), 295. https://doi.org/10.1007/s42979-021-00687-5
7. Chen, Z., Xu, Y., Zhang, C., & Tang, M. (2024). Prediction of glass chemical composition and type identification based on machine learning algorithms. Applied Sciences, 14(10), 4017. https://doi.org/10.3390/app14104017
8. Cover, T., & Hart, P. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21–27. https://doi.org/10.1109/TIT.1967.1053964

9. Deza, M. M., & Deza, E. (2016). Encyclopedia of distances (4th ed.). Springer. https://doi.org/10.1007/978-3-662-52844-0
10. German, B. (1987). Glass identification dataset. UCI Machine Learning Repository. https://doi.org/10.24432/C5WW2P
11. Hussien, A. G., Oliva, D., Houssein, E. H., Juan, A. A., & Yu, X. (2020). Binary whale optimization algorithm for dimensionality reduction. Mathematics, 8(10), 1821. https://doi.org/10.3390/math8101821
12. Karaboga, D. (2005). An idea based on honey bee swarm for numerical optimization (Technical Report TR06). Erciyes University, Engineering Faculty, Computer Engineering Department. https://abc.erciyes.edu.tr/pub/tr06_2005.pdf
13. Kaspi, O., Israelsohn-Azulay, O., Yigal, Z., Rosengarten, H., Krmpotić, M., Gouasmia, S., Radović, I. B., Jalkanen, P., Liski, A., Mizohata, K., Räisänen, J., Kasztovszky, Z., Harsányi, I., Acharya, R., Pujari, P. K., Mihály, M., Braun, M., Shabi, N., Girshevitz, O., & Senderowitz, H. (2023). Toward developing techniques—Agnostic machine learning classification models for forensically relevant glass fragments. Journal of Chemical Information and Modeling, 63(1), 87–100. https://doi.org/10.1021/acs.jcim.2c01362
14. Khan, M. A., Mazhar, T., Yaqoob, M. M., Khan, M. B., Saudagar, A. K. J., Ghadi, Y. Y., Khattak, U. F., & Shahid, M. (2024). Optimal feature selection for heart disease prediction using modified artificial bee colony (M-ABC) and k-nearest neighbors (KNN). Scientific Reports, 14, 26241. https://doi.org/10.1038/s41598-024-78021-1
15. Kirkpatrick, S., Gelatt, C. D., Jr., & Vecchi, M. P. (1983). Optimization by simulated annealing. Science, 220(4598), 671–680. https://doi.org/10.1126/science.220.4598.671
16. Li, G., Shi, M., Hao, T., Wang, P., Ma, Y., & Han, Y. (2024). A hybrid framework for ancient glass classification and recognition: Combining binary nutcracker optimization algorithm with KNN method. In 2024 9th International Conference on Control and Robotics Engineering (ICCRE) (pp. 345–351). IEEE. https://doi.org/10.1109/ICCRE61448.2024.10589876
17. Rachmatullah, M. I. C. (2022). The application of repeated SMOTE for multi-class classification on imbalanced data. MATRIK: Jurnal Manajemen, Teknik Informatika ve Rekayasa Komputer, 22(1), 13–24. https://doi.org/10.30812/matrik.v22i1.1803
18. Russell, S. J., & Norvig, P. (2020). Artificial intelligence: A modern approach (4th ed.). Pearson.
19. Schiezaro, M., & Pedrini, H. (2013). Data feature selection based on artificial bee colony algorithm. EURASIP Journal on Image and Video Processing, 2013, 47. https://doi.org/10.1186/1687-5281-2013-47
20. Uzer, M. S., Yilmaz, N., & Inan, O. (2013). Feature selection method based on artificial bee colony algorithm and support vector machines for medical datasets classification. The Scientific World Journal, 2013, 419187. https://doi.org/10.1155/2013/419187
21. Visweshwaran, S., Anbazhagan, M., & Ganesh, K. (2024). An empirical study on ML models with glass classification dataset. In H. Sharma, V. Shrivastava, A. K. Tripathi, & L. Wang (Eds.), Communication and Intelligent Systems. ICCIS 2023. Lecture Notes in Networks and Systems (Vol. 968). Springer. https://doi.org/10.1007/978-981-97-2079-8_30
22. Wang, Z., Lu, Y., & Li, T. (2023). Ancient glass classification based on random forest algorithm and decision tree model. Highlights in Science, Engineering and Technology, 34, 344–351. https://doi.org/10.54097/hset.v34i.5492