An Analysis of Intelligent Turkish Text Classification Models for Routing Calls in Call Centers: A Case Study on the Republic of Turkiye Ministry of Trade Call Center

Abstract

Call centers play a key role in the management of customer relationships in the modern business world. However, the growing demand for their services presents significant challenges, particularly in terms of staffing and handling increasing call volumes. This paper addresses these issues by presenting an AI-driven text classification framework tailored for the Republic of Turkiye Ministry of Trade Call Centre (MTCC), with the aim of automatically routing calls to relevant departments. Using a specific dataset of 20,000 phone call texts collected from the MTCC, the study employs TF-IDF, Word2Vec, and GloVe text vectorization techniques and applies various machine learning algorithms such as K-Nearest Neighbours, Naive Bayes, Support Vector Machines, Adaptive Boosting, Decision Tree and Random Forest for text classification. Through a comprehensive analysis, the study answers key research questions regarding optimal classifiers and vectorization methods. The proposed solution not only improves the efficiency of MTCC's call routing but also provides researchers with practical insights regarding Turkish text classification. The results indicate that a combination of the Random Forest classifier and Word2Vec text vectorization method is the optimal model that can manage to route calls in real-time.

Keywords

• Text classification
• Word2Vec
• GloVe
• TF-IDF
• Call centers

References

1. [1] P. G. Patterson, L. W. Johnson, and R. A. Spreng, “Modeling the Determinants of Customer Satisfaction for Business-to-Business Professional Services,” J. Acad. Mark. Sci., vol. 25, no. 1, pp. 4–17, 1996, doi: 10.1177/0092070397251002.
2. [2] V. Pallotta, R. Delmonte, L. Vrieling, and D. Walker, “Interaction Mining: The new frontier of Call Center Analytics,” in Proc. CEUR Workshop in DART@ AI* IA., vol. 771, Sep. 2011, pp. 1-12.
3. [3] Y. Park and S. C. Gates, “Towards real-time measurement of customer satisfaction using automatically generated call transcripts,” in Proc. Int. Conf. Inf. Knowl. Manag., vol. 24754, 2009, pp. 1387–1396, doi: 10.1145/1645953.1646128.
4. [4] S. A. Chowdhury, E. A. Stepanov, and G. Riccardi, “Predicting user satisfaction from turn-taking in spoken conversations,” presented at the INTERSPEECH, San Francisco, USA, Sept. 8-12, 2016, pp. 2910–2914, doi: 10.21437/Interspeech.2016-859.
5. [5] J. Luque, C. Segura, A. Sanchez, M. Umbert, and L. A. Galindo, “The role of linguistic and prosodic cues on the prediction of self-reported satisfaction in contact centre phone calls,” presented at the INTERSPEECH, Stockholm, Sweden, Aug. 20-24, 2017, pp. 2346–2350, doi: 10.21437/Interspeech.2017-424.
6. [6] J. Chatterjee, A. Saxena, and G. Vyas, “An automatic and robust system for identification of problematic call centre conversations,” presented at the Int. Conf. Micro-Electronics Telecommun. Eng. (ICMETE), Ghaziabad, India, Sept. 22-23, 2016, pp. 325–330, doi: 10.1109/ICMETE.2016.48.
7. [7] S. Meinzer, U. Jensen, A. Thamm, J. Hornegger, and B. M. Eskofier, “Can machine learning techniques predict customer dissatisfaction? A feasibility study for the automotive industry,” Artif. Intell. Res., vol. 6, no. 1, p. 80-90, Dec. 2016, doi: 10.5430/air.v6n1p80.
8. [8] Y. Liu, B. Cao, K. Ma, and J. Fan, “Improving the classification of call center service dialogue with key utterences,” Wirel. Networks, vol. 27, no. 5, pp. 3395–3406, 2021, doi: 10.1007/s11276-021-02573-7.

9. [9] S. Busemann, S. Schmeier, and R. G. Arens, “Message classification in the call center”, in Proc. Sixth Applied Natural Language Processing Conference, 2000, pp. 158–165, doi: 10.3115/974147.974169.
10. [10] D. Galanis, S. Karabetsos, M. Koutsombogera, H. Papageorgiou, A. Esposito, and M. T. Riviello, “Classification of emotional speech units in call centre interactions,” in Proc. 4th IEEE Int. Conf. Cogn. Infocommunications (CogInfoCom). Proc., 2013, pp. 403–406, doi: 10.1109/CogInfoCom.2013.6719279.
11. [11] E. P. Emmanuela, F. K. Tjendra, S. Kezia and D. Suryani, "Classification of Customer Satisfaction in Marketplace," presented at the 2023 International Conference on Computer Science, Information Technology and Engineering (ICCoSITE), Jakarta, ID, Feb. 16, 2023, doi: 10.1109/ICCoSITE57641.2023.10127788.
12. [12] A. Mousavi, M. Rezaee, and R. Ayanzadeh, “A survey on compressive sensing: Classical results and recent advancements,” J. Math. Model., vol. 8, no. 3, pp. 309–344, 2020, doi: 10.22124/jmm.2020.16701.1450.
13. [13] J. Salminen, M. Hopf, S. A. Chowdhury, S. gyo Jung, H. Almerekhi, and B. J. Jansen, “Developing an online hate classifier for multiple social media platforms,” Human-centric Comput. Inf. Sci., vol. 10, no. 1, pp. 1–34, Jan. 2020, doi: 10.1186/s13673-019-0205-6.
14. [14] R. L. Alaoui and E. H. Nfaoui, “Web attacks detection using stacked generalization ensemble for LSTMs and word embedding,” in Proc. Comput. Sci., vol. 215, 2022, pp. 687–696, doi: 10.1016/j.procs.2022.12.070.
15. [15] D. E. Cahyani and I. Patasik, “Performance comparison of tf-idf and word2vec models for emotion text classification,” Bull. Electr. Eng. Informatics, vol. 10, no. 5, pp. 2780–2788, 2021, doi: 10.11591/eei.v10i5.3157.
16. [16] S. Akuma, T. Lubem, and I. T. Adom, “Comparing Bag of Words and TF-IDF with different models for hate speech detection from live tweets,” Int. J. Inf. Technol., vol. 14, no. 7, pp. 3629–3635, 2022, doi: 10.1007/s41870-022-01096-4.
17. [17] B. C. ÖĞE and F. KAYAALP, “Farklı Sınıflandırma Algoritmaları ve Metin Temsil Yöntemlerinin Duygu Analizinde Performans Karşılaştırılması”, DUBİTED, vol. 9, no. 6, pp. 406–416, 2021, doi: 10.29130/dubited.1015320.
18. [18] B. Ekici and H. Takcı, “Spam Tespitinde Word2Vec ve TF-IDF Yöntemlerinin Karşılaştırılması ve Başarı Oranının Artırılması Üzerine Bir Çalışma,” Bilecik Şeyh Edebali Üniversitesi Fen Bilim. Derg., vol. 8, no. 2, pp. 646–655, 2021, doi: 10.35193/bseufbd.935247.
19. [19] K. Koruyan and A. Ekeryılmaz, “Makine Öğrenmesi ile Müşteri Şikayetlerinin Sınıflandırılması,” AJIT-e Acad. J. Inf. Technol., vol. 13, no. 50, pp. 168–183, 2022, doi: 10.5824/ajite.2022.03.004.x.
20. [20] Ö. ÇELİK and B. C. KOÇ, “TF-IDF, Word2vec ve Fasttext Vektör Model Yöntemleri ile Türkçe Haber Metinlerinin Sınıflandırılması”, DEUFMD, vol. 23, no. 67, pp. 121–127, 2021, doi: 10.21205/deufmd.2021236710.
21. [21] H. Saif, M. Fernandez, Y. He, and H. Alani, “On stopwords, filtering and data sparsity for sentiment analysis of twitter,” in Proc. 9th Int. Conf. Lang. Resour. Eval. Lr., 2014, pp. 810–817.
22. [22] C. Silva and B. Ribeiro, “The Importance of Stop Word Removal on Recall Values in Text Categorization,” in Proc. Int. Jt. Conf. Neural Networks, vol. 3, Aug. 2003, pp. 1661–1666, doi: 10.1109/ijcnn.2003.1223656.
23. [23] Y. Fan, C. Arora, and C. Treude, “Stop Words for Processing Software Engineering Documents: Do they Matter?,” in Proc. IEEE/ACM 2nd Int. Work. Nat. Lang. Softw. Eng. (NLBSE), 2023, pp. 40–47, doi: 10.1109/NLBSE59153.2023.00016.
24. [24] G. Gupta and S. Malhotra, "Text Document Tokenization for Word Frequency Count using Rapid miner," Int. J. Comput. Appl., vol.12, pp. 24-26, Aug. 2015.
25. [25] T. Korenius, J. Laurikkala, K. Järvelin, and M. Juhola, “Stemming and lemmatization in the clustering of finnish text documents,” in Proc. Int. Conf. Inf. Knowl. Manag., 2004, pp. 625–633, doi: 10.1145/1031171.1031285.
26. [26] A. Barbaresi, “Simplemma”. Zenodo, Jan. 20, 2023. doi: 10.5281/zenodo.7555188.
27. [27] W. Aljedaani et al., “Sentiment analysis on Twitter data integrating TextBlob and deep learning models: The case of US airline industry,” Knowledge-Based Syst., vol. 255, 2022, Art. no. 109780, doi: 10.1016/j.knosys.2022.109780.
28. [28] Mikolov, T., Chen, K., Corrado, G., & Dean, J., "Efficient Estimation of Word Representations in Vector," in Proc. 1st International Conference on Learning Representations, 2013, pp.1-12.
29. [29] A. K. Singh and M. Shashi, “Vectorization of text documents for identifying unifiable news articles,” Int. J. Adv. Comput. Sci. Appl., vol. 10, no. 7, pp. 305–310, 2019, doi: 10.14569/ijacsa.2019.0100742.
30. [30] G. Yeşiltaş and T. Güngör, "Intrinsic and Extrinsic Evaluation of Word Embedding Models," in Proc. Innovations in Intelligent Systems and Applications Conference (ASYU), 2020, pp. 1-6, doi: 10.1109/ASYU50717.2020.9259855.
31. [31] D. Jatnika, M. A. Bijaksana, and A. A. Suryani, “Word2vec model analysis for semantic similarities in English words,” in Proc. Comput. Sci., vol. 157, Sept.. 2019, pp. 160–167, doi: 10.1016/J.PROCS.2019.08.153.
32. [32] J. Pennington, R. Socher, and C. D. Manning, “GloVe: Global vectors for word representation,” in Proc. Conference on Empirical Methods in Natural Language Processing, Oct. 2014, pp. 1532–1543. doi: 10.3115/v1/d14-1162.
33. [33] T. M. Cover and P. E. Hart, “Nearest Neighbor Pattern Classification,” IEEE Trans. Inf. Theory, vol. 13, no. 1, pp. 21–27, 1967, doi: 10.1109/TIT.1967.1053964.
34. [34] J. Lakoumentas, J. Drakos, M. Karakantza, G. Sakellaropoulos, V. Megalooikonomou, and G. Nikiforidis, “Optimizations of the naïve-Bayes classifier for the prognosis of B-Chronic Lymphocytic Leukemia incorporating flow cytometry data,” Comput. Methods Programs Biomed., vol. 108, no. 1, pp. 158–167, 2012, doi: 10.1016/j.cmpb.2012.02.009.
35. [35] C. Cortes and V. Vapnik, “Support-Vector Networks,” Mach. Learn., vol. 20, no. 3, pp. 273–297, 1995, doi: 10.1023/A:1022627411411.
36. [36] S. METLEK and K. KAYAALP, “Derin Öğrenme ve Destek Vektör Makineleri İle Görüntüden Cinsiyet Tahmini”, DUBİTED, vol. 8, no. 3, pp. 2208–2228, 2020, doi: 10.29130/dubited.707316.
37. [37] M. Kantardzic, “Decision Trees and Decision Rules,” in Data Mining: Concepts, Models, Methods, and Algorithms, 3rd ed. Columbia, MD, U.S.A.: Wiley-IEEE Press, 2019, sec. 6, pp. 197-229
38. [38] R. Wang, “AdaBoost for Feature Selection, Classification and Its Relation with SVM, A Review,” in Proc. Phys. Procedia, vol. 25, 2012, pp. 800–807, doi: 10.1016/j.phpro.2012.03.160.
39. [39] E. Scornet, G. Biau, and J. P. Vert, “Consistency of random forests,” Ann. Stat., vol. 43, no. 4, pp. 1716–1741, Aug. 2015, doi: 10.1214/15-AOS1321.
40. [40] P. Baldi, S. Brunak, Y. Chauvin, C. A. F. Andersen, and H. Nielsen, “Assessing the accuracy of prediction algorithms for classification: An overview,” Bioinformatics, vol. 16, no. 5, pp. 412–424, May 2000, doi: 10.1093/bioinformatics/16.5.412.