A Machine Learning-Based Automatic E-mail Classification System for Corporate Complaint Management

Öz

Manual management of complaint emails following corporate gifting processes, specifically the distribution of employee benefits such as Ramadan or New Year packages, creates a significant operational burden for procurement departments. This study focuses on this specific problem and aims to develop an artificial intelligence (AI)-supported, multi-label system for the automatic classification and management of complaint emails. The system classifies complaints into four main categories: product type, package type, complaint type, and responsible person. In the study, to overcome privacy constraints associated with real-world corporate data, a unique dataset containing 1,870 synthetic complaint emails was generated using large language models based on actual operational scenarios. Texts were vectorized using the TF-IDF method, and the performance of ten machine learning algorithms, including CatBoost, XGBoost, and Support Vector Machines (SVM), was evaluated. The findings revealed that the developed system achieved an average F1-score of 87.75%. Distinguishing itself from the extensive literature primarily focused on spam and phishing detection, this study offers a unique contribution by applying the SHAP (SHapley Additive exPlanations) method to the domain-specific problem of multi-label corporate complaint management. In this way, trust in the system is increased by explaining which words the model predictions are based on. The developed explainable AI-supported system offers an effective and interpretable solution for increasing automation and efficiency in complaint management processes.

Anahtar Kelimeler

• Email Classification
• Explainable Artificial Intelligence
• Synthetic Data Generation
• Corporate Complaint Management
• Multi-label Classification

Kurumsal Şikayet Yönetimi için Makine Öğrenmesi Tabanlı Otomatik E-posta Sınıflandırma Sistemi

Öz

Kurumsal hediyeleşme süreçleri sonrasında, özellikle çalışanlara yan hak olarak sunulan Ramazan, yılbaşı gibi hediye paketlerinin dağıtımının ardından gelen şikayet e-postalarının manuel yönetimi, satın alma departmanları için ciddi bir operasyonel yük oluşturmaktadır. Bu çalışma, bu spesifik probleme odaklanarak, şikayet e-postalarının otomatik sınıflandırılması ve yönetimi için yapay zekâ (YZ) destekli, çok etiketli bir sistem geliştirmeyi amaçlamaktadır. Sistem, şikayetleri; ürün türü, paket türü, şikâyet türü ve sorumlu kişi olmak üzere dört ana kategoride sınıflandırmaktadır. Çalışmada, gerçek şirket verilerinin gizlilik kısıtları nedeniyle, büyük dil modelleri kullanılarak ve gerçek operasyonel senaryolar temel alınarak 1.870 adet sentetik şikâyet e-postası içeren özgün bir veri seti oluşturulmuş ve TF-IDF yöntemiyle metinler vektörleştirilmiştir. CatBoost, XGBoost ve Destek Vektör Makineleri (SVM) dâhil olmak üzere on farklı makine öğrenmesi algoritmasının performansı karşılaştırmalı olarak değerlendirilmiştir. Elde edilen bulgular, geliştirilen sistemin %87.75 ortalama F1-skoru başarısı gösterdiğini ortaya koymuştur. Literatürdeki spam ve oltalama odaklı çalışmaların aksine, bu çalışmanın özgün katkısı; çok etiketli kurumsal şikayet yönetimi problemine SHAP (SHapley Additive exPlanations) yöntemini uygulayarak model kararlarını şeffaf hale getirmesidir. Bu sayede, model tahminlerinin hangi kelimelere dayandığı açıklanarak sisteme duyulan güven artırılmıştır. Geliştirilen bu açıklanabilir YZ destekli sistem, kurumsal hediye paketleriyle ilgili şikayet yönetimi süreçlerinde otomasyon ve verimlilik artışı için etkin ve yorumlanabilir bir çözüm sunmaktadır.

Anahtar Kelimeler

• Kurumsal Şikayet Yönetimi
• Çok Etiketli Sınıflandırma
• Açıklanabilir Yapay Zekâ
• Sentetik Veri Üretimi
• E-Posta Sınıflandırma

Kaynakça

1. Adadi, A., Berrada, M., 2018. Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access, 6, 52138-52160.
2. Brutlag, J.D., Meek, C., 2000. Challenges of the email domain for text classification. Proceedings of the Seventeenth International Conference on Machine Learning, pp. 103-110.
3. Cai, H., Shao, X., Zhou, P., Li, H. (2025). Multi-Label Classification of Complaint Texts: Civil Aviation Service Quality Case Study. Electronics, 14(3), 434.
4. Chen, T., Guestrin, C., 2016. XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794.
5. Cohen, W.W., 1996. Learning rules that classify e-mail. Proceedings of the 1996 AAAI Spring Symposium on Information Access, p. 25.
6. Cortes, C., Vapnik, V., 1995. Support-vector networks. Machine Learning, 20(3), 273–297.
7. Dai, H., Liu, Z., Liao, W., Huang, X., Cao, Y., Wu, Z., ... & Li, X. (2025). Auggpt: Leveraging chatgpt for text data augmentation. IEEE Transactions on Big Data.
8. Deniz, E., Erbay, H., & Coşar, M. (2022). Multi-label classification of e-commerce customer reviews via machine learning. Axioms, 11(9), 436.

9. Dewi, C., Tsai, B. J., & Chen, R. C. (2022, November). Shapley additive explanations for text classification and sentiment analysis of internet movie database. In Asian Conference on Intelligent Information and Database Systems (pp. 69-80). Singapore: Springer Nature Singapore.
10. Fung, H.P., 2014. Criteria, use cases, and effects of information technology process automation (ITPA). Advances in Robotics & Automation, 3(3), 1-11.
11. Gopali, S., Abri, F., Namin, A. S., & Jones, K. S. (2024). The Applicability of LLMs in Generating Textual Samples for Analysis of Imbalanced Datasets. IEEE Access.
12. Goodfellow, I., Bengio, Y., Courville, A., 2016. Deep learning. The MIT Press.
13. Kiritchenko, S., & Matwin, S. (2001). Email classification with co-training. Proceedings of the 2001 Conference of the Centre for Advanced Studies on Collaborative Research, p. 8.
14. Kokalj, E., Škrlj, B., Lavrač, N., Pollak, S., & Robnik-Šikonja, M. (2021, April). BERT meets shapley: Extending SHAP explanations to transformer-based classifiers. In Proceedings of the EACL hackashop on news media content analysis and automated report generation (pp. 16-21).
15. Kuyucuk, T., & Çallı, L. (2022). Using multi-label classification methods to analyze complaints against cargo services during the COVID-19 outbreak: Comparing survey-based and word-based labeling. Sakarya University Journal of Computer and Information Sciences, 5(3), 371-384.
16. Lundberg, S.M., Lee, S.I., 2017. A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems 30, pp. 4765–4774.
17. Mitchell, T.M., 1997. Machine learning. McGraw-Hill.
18. Mosca, E., Szigeti, F., Tragianni, S., Gallagher, D., & Groh, G. (2022, October). SHAP-based explanation methods: a review for NLP interpretability. In Proceedings of the 29th international conference on computational linguistics (pp. 4593-4603).
19. Mujtaba, G., Shuib, L., Raj, R.G., Majeed, N., Al-Garadi, M.A., 2017. Email Classification Research Trends: A Review and Open Issues. IEEE Access, 5, 9044–9064.
20. Patwardhan, N., Marrone, S., Sansone, C., 2023. Transformers in the real world: A survey on NLP applications. Information, 14(4), 242.
21. Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V., Gulin, A., 2018. CatBoost: Unbiased boosting with categorical features. Advances in Neural Information Processing Systems 31, pp. 6638–6648.
22. Türkmen, A., Bayram, B., & Aydın, G. (2023, July). Employee behavior analysis towards multi-label classification of customer reviews. In Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments (pp. 511-517).
23. ValizadehAslani, T., Shi, Y., Wang, J., Ren, P., Zhang, Y., Hu, M., ... & Liang, H. (2024). Two-stage fine-tuning with ChatGPT data augmentation for learning class-imbalanced data. Neurocomputing, 592, 127801.
24. Vilone, G., Longo, L., 2021. Notions of explainability and evaluation approaches for explainable artificial intelligence. Information Fusion, 76, 89–106.
25. Willcocks, L., Lacity, M., 2016. A new approach to automating services. MIT Sloan Management Review, 58(1), 40–49.
26. Zhao, W., Joshi, T., Nair, V. N., & Sudjianto, A. (2020). Shap values for explaining cnn-based text classification models. arXiv preprint arXiv:2008.11825.