Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach

Umut Saray

doi:10.5281/zenodo.23618340

TR EN

Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach

Öz

In this study, Random Forest, XGBoost, SVM, and MLP classifiers were compared for machine failure prediction on the AI4I 2020 Predictive Maintenance dataset under two class-imbalance handling scenarios: SMOTE-based oversampling and class weighting. All preprocessing steps were applied leakage-free (within cross-validation folds), model selection was based on training CV PR-AUC, and the difference between models was tested using a Holm-corrected 5x2cv statistical significance test. Selected as the final model according to the pre-specified training CV PR-AUC criterion, XGBoost with class weighting achieved 0,9600 accuracy, 0,8529 recall, 0,9702 ROC-AUC, and 0,7673 PR-AUC on the test set; only one of 28 pairwise comparisons was statistically significant. SHAP analysis identified tool wear as the feature contributing most strongly to model predictions. Results are compared against high-accuracy claims in the literature; findings support the importance of a leakage-free evaluation protocol.

Anahtar Kelimeler

predictive maintenance, machine learning, class imbalance, explainable AI, SHAP

Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach

Öz

In this study, Random Forest, XGBoost, SVM, and MLP classifiers were compared for machine failure prediction on the AI4I 2020 Predictive Maintenance dataset under two class-imbalance handling scenarios: SMOTE-based oversampling and class weighting. All preprocessing steps were applied leakage-free (within cross-validation folds), model selection was based on training CV PR-AUC, and the difference between models was tested using a Holm-corrected 5x2cv statistical significance test. Selected as the final model according to the pre-specified training CV PR-AUC criterion, XGBoost with class weighting achieved 0,9600 accuracy, 0,8529 recall, 0,9702 ROC-AUC, and 0,7673 PR-AUC on the test set; only one of 28 pairwise comparisons was statistically significant. SHAP analysis identified tool wear as the feature contributing most strongly to model predictions. Results are compared against high-accuracy claims in the literature; findings support the importance of a leakage-free evaluation protocol.

Anahtar Kelimeler

predictive maintenance, machine learning, class imbalance, explainable AI, SHAP

Kaynakça

[1] S. Matzka, "Explainable Artificial Intelligence for Predictive Maintenance Applications," in Proc. 2020 IEEE Third Int. Conf. Artificial Intelligence for Industries (AI4I), Irvine, CA, USA, 2020, pp. 69-74, doi: 10.1109/AI4I49448.2020.00023.
[2] S. Matzka, "AI4I 2020 Predictive Maintenance Dataset," UCI Machine Learning Repository, 2020. [Online]. doi: 10.24432/C5HS5C.
[3] W. Zhao, J. Ding, X. Huang, and Y. Zhang, "Research on Milling Machine Predictive Maintenance Based on Machine Learning and SHAP Analysis in Intelligent Manufacturing Environment," in Proc. 2025 IEEE 15th Int. Conf. Electron-ics Information and Emergency Communication (ICEIEC), 2025, doi: 10.1109/ICEIEC65904.2025.11273156.
[4] H. Taoufyq, K. El Guemmat, K. Mansouri, and F. Akef, "A Hybrid Predictive Maintenance Approach for Optimizing Industrial Performance in Industry 4.0," International Jour-nal of Intelligent Engineering and Systems, 2026, doi: 10.22266/ijies2026.0531.32.
[5] N. Makesh, M. Naveenkumar, J. Kavitha, and T. Chankyan, "Prognostix-XAI: Interpretable Modeling of Mechanical Degradation Dynamics," in Proc. 4th Int. Conf. Intelligent Data Communication Technologies and Internet of Things (IDCIoT), 2026, doi: 10.1109/IDCIoT67589.2026.11455807.
[6] J. R. D. Luche, B. G. C. D. S. Goussain, and C. R. D. de Freitas, "Robust Baselines and Probability Calibration for TPM-Oriented Predictive Maintenance," International Jour-nal of Prognostics and Health Management, 2026, doi: 10.36001/IJPHM.2026.v17i1.4659.
[7] M. Alnahhal, M. I. Tabash, S. K. Safi, and Z. Mamadiyarov, "A Comparative Study of Imbalance-Handling Methods in Multiclass Predictive Maintenance," Computation, 2026, doi: 10.3390/computation14040088.
[8] H. Azzaoui, "Ensemble learning for safety-critical fault detec-tion in industrial distillation: A comprehensive benchmark with noise robustness analysis," Mechanical Systems and Signal Processing, 2026, doi: 10.1016/j.ymssp.2026.114599.
[9] A. H. Hussein, S. B. A. Anas, S. Yaakob, and Y. I. Go, "Ma-chine learning approaches for classical optical time-domain reflectometer: A review," Optical Switching and Network-ing, 2026, doi: 10.1016/j.osn.2026.100880.
[10] J. Dawkrajai, W. Weerachapichasgul, W. Daosud, and P. Kittisupakorn, "Enhancing Fault Diagnosis in Imbalanced Data Using Weighted GRU Architecture," Engineering Journal, vol. 29, no. 7, 2025, doi: 10.4186/ej.2025.29.7.35.

[11] S. M. Lundberg and S.-I. Lee, "A unified approach to inter-preting model predictions," in Advances in Neural Infor-mation Processing Systems 30 (NeurIPS 2017), 2017.
[12] M. T. Ribeiro, S. Singh, and C. Guestrin, "'Why should I trust you?': Explaining the predictions of any classifier," in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2016, pp. 1135-1144, doi: 10.1145/2939672.2939778.
[13] M. Poliak, L. Pawlik, and D. Frej, "Explainable Deep Learn-ing for Bearing Fault Diagnosis: Architectural Superiority of ResNet-1D Validated by SHAP," Electronics, vol. 14, no. 24, 2025, doi: 10.3390/electronics14244875.
[14] A. P. Chowdhury, F. N. Nur, A. H. M. S. Islam, and M. A. Shah, "FLEX-IDS: A secure and explainable federated in-trusion detection framework for Edge-of-Things environ-ments under adversarial conditions," Computers and Elec-trical Engineering, 2026, doi: 10.1016/j.compeleceng.2025.110827.
[15] M. L. Ndao, G. Youness, N. Niang, and G. Saporta, "Im-proving predictive maintenance: Evaluating the impact of preprocessing and model complexity on the effectiveness of eXplainable Artificial Intelligence methods," Engineering Applications of Artificial Intelligence, 2025, doi: 10.1016/j.engappai.2025.110144.
[16] S. Gawde, S. Patil, S. Kumar, and S. Alfarhood, "Explaina-ble Predictive Maintenance of Rotating Machines Using LIME, SHAP, PDP, ICE," IEEE Access, 2024, doi: 10.1109/ACCESS.2024.3367110.
[17] Z. Çetinkaya, F. Horasan, H. Aydilek, and M. Y. Erten, "Predictive Modeling for Milk Quality Using Machine Learning and XAI Algorithms," ADBA Computer Science, vol. 3, no. 1, pp. 63-71, 2026, doi: 10.69882/adba.cs.2026019.
[18] A. Hamdi Alaoui and A. Meddaoui, "Imbalance-Aware Predictive Maintenance Framework with Explainable Ma-chine Learning: A case study using the AI4I 2020 Dataset," in Proc. 2026 6th Int. Conf. Innovative Research in Applied Science, Engineering and Technology (IRASET), 2026, doi: 10.1109/IRASET68627.2026.11538682.
[19] B. Ghasemkhani, R. A. Kut, D. Birant, and R. Yilmaz, "Bal-anced Hoeffding Tree Forest (BHTF): A Novel Multi-Label Classification with Oversampling and Undersampling Tech-niques for Failure Mode Diagnosis in Predictive Mainte-nance," Mathematics, vol. 13, no. 18, art. 3019, 2025, doi: 10.3390/math13183019.
[20] B. Ghasemkhani, O. Aktaş, and D. Birant, "Balanced K-Star: An Explainable Machine Learning Method for Inter-net-of-Things-Enabled Predictive Maintenance in Manufac-turing," Machines, 2023, doi: 10.3390/machines11030322.
[21] A. Hatipoglu, Y. Güneri, and E. Yılmaz, "A comparative predictive maintenance application based on machine and deep learning," Journal of the Faculty of Engineering and Architecture of Gazi University, 2024, doi: 10.17341/gazimmfd.1221105.
[22] A. Narin, N. Bektaşoğlu, and U. Ileri, "Machine learning applications in predictive maintenance: Data balancing and feature selection analysis," Journal of the Faculty of Engi-neering and Architecture of Gazi University, 2026, doi: 10.17341/gazimmfd.1676005.
[23] T. G. Dietterich, "Approximate statistical tests for comparing supervised classification learning algorithms," Neural Com-putation, vol. 10, no. 7, pp. 1895-1923, 1998, doi: 10.1162/089976698300017197.
[24] A. Nadarajan, I. Ishak, N. Manshor, and M. Y. Yusof, "En-hancing Predictive Maintenance Method Using Machine Learning to Improve IoT-Embedded Machinery Efficiency and Performance," International Journal of Advanced Com-puter Science and Applications, 2025, doi: 10.14569/IJACSA.2025.0161027.

Ayrıntılar

Birincil Dil

İngilizce

Konular

Mekatronik Mühendisliği

Bölüm

Araştırma Makalesi

Yazarlar

Umut Saray ^*
0000-0003-3339-6876
Türkiye

Yayımlanma Tarihi

29 Ekim 2026

Gönderilme Tarihi

5 Eylül 2026

Kabul Tarihi

29 Eylül 2026

Yayımlandığı Sayı

Yıl 2026 Cilt: 3 Sayı: 2

DOI

https://doi.org/10.5281/zenodo.23618340

IZ

https://izlik.org/JA74BD24FF

APA

Saray, U. (2026). Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach. Hendese Teknik Bilimler ve Mühendislik Dergisi, 3(2), 38-45. https://doi.org/10.5281/zenodo.23618340

AMA

1.Saray U. Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach. HENDESE. 2026;3(2):38-45. doi:10.5281/zenodo.23618340

Chicago

Saray, Umut. 2026. “Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach”. Hendese Teknik Bilimler ve Mühendislik Dergisi 3 (2): 38-45. https://doi.org/10.5281/zenodo.23618340.

EndNote

Saray U (01 Ekim 2026) Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach. Hendese Teknik Bilimler ve Mühendislik Dergisi 3 2 38–45.

IEEE

[1]U. Saray, “Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach”, HENDESE, c. 3, sy 2, ss. 38–45, Eki. 2026, doi: 10.5281/zenodo.23618340.

ISNAD

Saray, Umut. “Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach”. Hendese Teknik Bilimler ve Mühendislik Dergisi 3/2 (01 Ekim 2026): 38-45. https://doi.org/10.5281/zenodo.23618340.

JAMA

1.Saray U. Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach. HENDESE. 2026;3:38–45.

MLA

Saray, Umut. “Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach”. Hendese Teknik Bilimler ve Mühendislik Dergisi, c. 3, sy 2, Ekim 2026, ss. 38-45, doi:10.5281/zenodo.23618340.

Vancouver

1.Umut Saray. Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach. HENDESE. 01 Ekim 2026;3(2):38-45. doi:10.5281/zenodo.23618340

Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach

Öz

Anahtar Kelimeler

Machine Failure Classification under Class Imbalance in Predictive Maintenance: A Comparative Machine Learning and SHAP‑Based Explainability Approach

Öz

Anahtar Kelimeler

Kaynakça

Ayrıntılar

Birincil Dil

Konular

Bölüm

Yazarlar

Yayımlanma Tarihi

Gönderilme Tarihi

Kabul Tarihi

Yayımlandığı Sayı

DOI

IZ

Kaynak Göster