Automatic Classification of Banking Branch Requests and Errors with Natural Language Processing and Machine Learning

Abstract

: In today’s world, the service sector is undergoing rapid technological development. Keeping pace with this transformation is not only a necessity for institutions but also essential for gaining a competitive advantage. Technological advancements have made it crucial to respond to customer demands quickly and accurately.This study emphasizes the importance of efficiently classifying customer demands and software errors in the branches of a bank operating in the finance sector. Real-world data was divided into three categories: Desktop Support, Software Support, and Field Support, with a total of 4,500 samples equally distributed among the categories. Eighty percent of the data was used for training and 20% for testing machine learning algorithms such as Bidirectional Encoder Representations from Transformers (BERT), Naive Bayes, Random Forest, and Artificial Neural Networks (ANN). The models were trained separately using CountVectorizer and Term Frequency–Inverse Document Frequency (TF-IDF) metrics. The dataset was also analyzed using two sample sizes: 3,000 and 4,500. The best results were obtained with BERT and ANN models using 4,500 samples and the TF-IDF metric, achieving accuracy rates above 92%. The positive effects of increased data size and the TF-IDF metric were evident. Additionally, ANN-based models proved more effective for this type of classification problem.

Keywords

• Machine Learning
• Classification
• Natural Language Processing (NLP)

References

1. [1] İlhan, N., & Sağaltıcı, D. (2020). Sentiment Analysis on Twitter. Harran University Engineering Journal, 5(2), 146-156. https://doi.org/10.46
2. [2] E. Kumaş, "Comparison of Classifiers in Sentiment Analysis from Turkish Twitter Data", ESTUDAM Bilişim, vol. 2, pp. 2, pp. 1-5, 2021.
3. [3] Yıldırım, S., & Yıldız, T. (2018). Comparative text classification analysis for Turkish. Pamukkale University Journal of Engineering Sciences, 24(5), 879-886.
4. [4] Toğaçar, M., Eşidir, K. A., & Ergen, B. (2022). Detection of Fake News Published on the Internet Using Artificial Intelligence-Based Natural Language Processing Approach. Journal of Intelligent Systems: Theory and Applications, 5(1), 1-8. https://doi.org/10.38016/jista.950713
5. [5] Kocak, S., İç, Y. T., Sert, M., Dengiz, B. (2023). Natural Turkish language processing based method for classification of R&D projects. Gazi University Journal of Engineering and Architecture Faculty, 38(3), 1375-1388. https://doi.org/10.17341/gazimmfd.889395
6. [6] Görentaş, M. B., Uçkan, T., & Bayram Arlı, N. (2023). Classification of Court of Dispute Decisions with Machine Learning Methods. Journal of Yüzüncü Yıl University Graduate School of Science and Technology, 28(3), 947-961. https://doi.org/10.53433/yyufbed.1292275
7. [7] W. Songpan, "The analysis and prediction of customer review rating using opinion mining," 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA), London, UK, 2017, pp. 71-77, doi: 10.1109/SERA.2017.7965709.
8. [8]Kaşıkçı, T., & Gökçen, H. (2014). Identification of E-Commerce Sites with Text Mining. Journal of Information Technologies, 7(1). https://doi.org/10.12973/bid.2014]

9. [9] Arslan, H., Dadaş, I. E., & Işık, Y. E. (2022). Demand Classification with Different Vectorisation and Preprocessing Methods. Düzce University Journal of Science and Technology, 10(3), 1433-1442. https://doi.org/10.29130/dubited.1017422
10. [10]S. Kazan and H. Karakoca, "Product Category Classification with Machine Learning", SAUCIS, vol. 2, pp. 1, pp. 18-27, 2019, doi: 10.35377/saucis.02.01.523139.
11. [11] Classification of Customer Complaints with Machine Learning* Kutan KORUYAN, Dokuz Eylül University, Faculty of Economics and Administrative Sciences, Department of Management Information Systems, Ph. Member, 0000-0002-3115-5676
12. [12] Tekin, M. C., & Tunalı, V. (2019). Prioritisation of software development requests with text mining methods. Pamukkale University Journal of Engineering Sciences, 25(5), 615-620.
13. [13] Aydemir, E., Işık, M., & Tuncer, T. (2021). Classification of Turkish News Texts by Using Multinomial Naive Bayes Algorithm. Fırat University Journal of Engineering Sciences, 33(2), 519-526. https://doi.org/10.35234/fumbd.871986
14. [14] Sevimli Deniz, S. (2021). Comparison of Rule-Based Classification Algorithms. Data Science, 4(3), 72-80.
15. [15] Binici, K. (2019). A Study on Automatic Assignment of Standard File Plan Numbers to e-Documents by Machine Learning Approach. Knowledge Management, 2(2), 116-126. https://doi.org/10.33721/by.654464
16. [16] H. Deng, Y. Sun, Y. Chang, J. Han, "Probabilistic Models for Classification" in C.C. Aggarwal (Eds.), Data Classification Algorithms and Applications (pp. 67-70), CRC Press, New York, USA, 2015.
17. [17] G. Louppe, "Understanding Random Forest", PhD thesis, University of Liege, 2015.
18. [18] J.M. Zurada, "Introduction to Artificial Neural Systems", West Publishing Company, 1992.
19. [19] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K., "Bert: Pre-trainingof deep bidirectional transformers for language understanding," arXivpreprint arXiv:1810.04805, 2018.
20. [20]Gupta, Shashij, et al. "Machine translation testing via pathological invariance." Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2020.
21. [21] Do, Quang-Minh, Kungan Zeng, and Incheon Paik. "Resolving Lexical Ambiguity in English-Japanese Neural Machine Translation." 2020 3rd Artificial Intelligence and Cloud Computing Conference. 2020.
22. [22]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez,A. N., Kaiser, L., and Polosukhin, I., "Attention is all you need," arXivpreprint arXiv:1706.03762, 2017