Relational Turkish Text Classification with Graph Neural Networks

Öz

Text classification and relational analysis in Turkish play a critical role in understanding the language's complex structure and enhancing natural language processing (NLP) procedures. This study focuses on the classification of Turkish texts and the in-depth analysis of the relationships between them. The aim of the study is to develop an advanced classification model that effectively captures the rich morphological structure of Turkish and the intertextual relationships. Using a dataset obtained from the TRT-Haber website, graph-based deep learning techniques were employed to create a high-performance model. The BERT (BertTurk) model was used for semantic vector representations of texts, and adjacency matrices representing intertextual relationships were integrated. These data were then fed into a graph neural network (GNN) based classification model. The results demonstrate that the GNN model can classify texts with a remarkable accuracy rate of 97.93% and successfully resolve relational structures. These findings highlight the effectiveness and potential of graph-based approaches in text classification and relational analysis, contributing to the development of innovative methods for better understanding and processing Turkish texts.

Anahtar Kelimeler

• Turkish NLP
• Relational text classification
• Deep learning
• Graph neural networks

Graf Sinir Ağları ile İlişkisel Türkçe Metin Sınıflandırma

Öz

Türkçe metin sınıflandırması ve ilişkisel analiz, dilin karmaşık yapısını anlamada ve doğal dil işleme süreçlerini geliştirmede kritik bir rol oynar. Bu çalışma, Türkçe metinlerin sınıflandırılması ve aralarındaki ilişkilerin derinlemesine analiz edilmesine odaklanmaktadır. Çalışmanın amacı, Türkçe'nin zengin morfolojik yapısını ve metinler arası ilişkileri etkin bir şekilde ele alarak, bu yapıyı yansıtan ileri düzey bir sınıflandırma modeli geliştirmektir. TRT-Haber web sayfasından elde edilen veri kümesi üzerinde graf tabanlı derin öğrenme teknikleri kullanılarak, yüksek performanslı bir model oluşturulmuştur. Metinlerin semantik vektör gösterimleri için BERT (BertTurk) modeli kullanılmış ve metinler arası ilişkileri gösteren kenar komşuluk matrisleri ile birleştirilmiştir. Bu veriler, graf sinir ağı (GNN) tabanlı sınıflandırma modeline beslenmiştir. Elde edilen sonuçlar, GNN modelinin %97.93 doğruluk oranı ile metinleri sınıflandırabildiğini ve ilişkisel yapıları başarıyla çözümleyebildiğini göstermektedir. Bu bulgular, metin sınıflandırması ve ilişkisel analizde graf tabanlı yaklaşımların etkinliğini ve potansiyelini ortaya koyarak, Türkçe metinlerin daha iyi anlaşılmasını ve işlenmesini sağlayacak yenilikçi yöntemlerin geliştirilmesine katkı sağlamaktadır.

Anahtar Kelimeler

• Türkçe doğal dil işleme
• İlişkisel metin sınıflandırma
• Derin öğrenme
• Graf sinir ağları

Kaynakça

1. [1] Dhar, A., Mukherjee, H., Dash, N. S. ve Roy, K., “Text categorization: past and present.”, Artificial Intelligence Review, 54, 3007-3054, (2021).
2. [2] Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., & Brown, D., “Text classification algorithms: A survey.”, Information, 10(4), 150, (2019).
3. [3] Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., & Gao, J., “Deep learning--based text classification: a comprehensive review.”, ACM computing surveys (CSUR), 54(3), 1-40, (2021).
4. [4] Büyük, O., “Joint intent detection and slot filling for Turkish natural language understanding.”, Turkish Journal of Electrical Engineering and Computer Sciences, 31(5), 844-859, (2023).
5. [5] Pan, D., Yang, Z., Tan, H., Wu, J., & Lin, H., “Dialogue topic extraction as sentence sequence labeling.”, In CCF International Conference on Natural Language Processing and Chinese Computing (pp. 252-262). Cham: Springer, (2022).
6. [6] Tohma, K., Okur, H. I., Kutlu, Y., & Sertbas, A., “Sentiment Analysis in Turkish Question Answering Systems: An Application of Human-Robot Interaction.”, IEEE Access, (2023).
7. [7] Nature Switzerland. Koru, G. K., & Uluyol, Ç., “Detection of Turkish Fake News from Tweets with BERT Models.”, IEEE Access, (2024).
8. [8] Karasoy, O., & Ballı, S., “Spam SMS detection for Turkish language with deep text analysis and deep learning methods.”, Arabian Journal for Science and Engineering, 47(8), 9361-9377, (2022).

9. [9] Çıtlak O., Dörterler M. ve Doğru İ. A., “A hybrid spam detection framework for social networks”, Journal of Polytechnic, 26(2): 823-837, (2023).
10. [10] Zucco, C., Calabrese, B., Agapito, G., Guzzi, P. H., & Cannataro, M., “Sentiment analysis for mining texts and social networks data: Methods and tools.”, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(1), e1333, (2020).
11. [11] Shivakumara, P., Alaei, A., & Pal, U., “Mining text from natural scene and video images: A survey.”, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 11(6), e1428, (2021).
12. [12] Pintas, J. T., Fernandes, L. A., & Garcia, A. C. B., “Feature selection methods for text classification: a systematic literature review.”, Artificial Intelligence Review, 54(8), 6149-6200, (2021).
13. [13] Okur, H.I., Tohma, K., Sertbas, A., “Relational turkish text classification using distant supervised entities and relations.”, Computers, Materials & Continua, 79(2), 2209-2228, (2024).
14. [14] Bilen B., Horasan F., “LSTM network based sentiment analysis for customer reviews”, Politeknik Dergisi, 25(3): 959-966, (2022).
15. [15] Tohma, K., & Kutlu, Y., “Challenges Encountered in Turkish Natural Language Processing Studies.”, Natural and Engineering Sciences, 5(3), 204-211, (2020).
16. [16] Gasparetto, A., Marcuzzo, M., Zangari, A., & Albarelli, A., “A survey on text classification algorithms: From text to predictions.” Information, 13(2), 83, (2022).
17. [17] Li, Q., Peng, H., Li, J., Xia, C., Yang, R., Sun, L., ... & He, L. “A survey on text classification: From traditional to deep learning.”, ACM Transactions on Intelligent Systems and Technology (TIST), 13(2), 1-41, (2022).
18. [18] Kumar, A. V., Gupta, D., & Venugopalan, M., “Cyberbullying Text Classification for Social Media Data Using Embedding and Deep Learning Approaches.”, In 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT) (pp. 1-6). IEEE, (2023).
19. [19] Türkmen, H., Dikenelli, O., Eraslan, C., Callı, M. C., & Özbek, S. S., “BioBERTurk: Exploring Turkish Biomedical Language Model Development Strategies in Low-Resource Setting.”, Journal of Healthcare Informatics Research, 7(4), 433-446, (2023).
20. [20] Umer, M., Imtiaz, Z., Ahmad, M., Nappi, M., Medaglia, C., Choi, G. S., & Mehmood, A., “Impact of convolutional neural network and FastText embedding on text classification.”, Multimedia Tools and Applications, 82(4), 5569-5585, (2023).
21. [21] Altinel Girgin, A. B., “Semantic text classification: A survey of past and recent advances.”, (2018).
22. [22] Zhang, Y., Yu, X., Cui, Z., Wu, S., Wen, Z., & Wang, L., “Every document owns its structure: Inductive text classification via graph neural networks.”, arXiv preprint arXiv:2004.13826, (2020).
23. [23] Nguyen, G., Dlugolinsky, S., Bobák, M., Tran, V., López García, Á., Heredia, I., ... & Hluchý, L., “Machine learning and deep learning frameworks and libraries for large-scale data mining: a survey.”, Artificial Intelligence Review, 52, 77-124, (2019).
24. [24] Cervetti, G. N., & Wright, T. S., “The role of knowledge in understanding and learning from text.”, Handbook of reading research, 5, (2020).
25. [25] Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., & Brown, D., “Text classification algorithms: A survey.”, Information, 10(4), 150, (2019).
26. [26] Wu, L., Chen, Y., Shen, K., Guo, X., Gao, H., Li, S., ... & Long, B., “Graph neural networks for natural language processing: A survey.”, Foundations and Trends® in Machine Learning, 16(2), 119-328, (2023).
27. [27] Uslu, O., & Özmen-akyol, S., “Türkçe haber metinlerinin makine öğrenmesi yöntemleri kullanılarak sınıflandırılması.”, Eskişehir Türk Dünyası Uygulama ve Araştırma Merkezi Bilişim Dergisi, 2(1), 15-20, (2021).
28. [28] Çelik, Ö., & Koç, B. C., “TF-IDF, Word2vec ve Fasttext vektör model yöntemleri ile Türkçe haber metinlerinin sınıflandırılması.”, Dokuz Eylül Üniversitesi Mühendislik Fakültesi Fen ve Mühendislik Dergisi, 23(67), 121-127, (2021).
29. [29] Acı, Ç., & Çırak, A., “Türkçe haber metinlerinin konvolüsyonel sinir ağları ve Word2Vec kullanılarak sınıflandırılması.”, Bilişim Teknolojileri Dergisi, 12(3), 219-228, (2019).
30. [30] Karakurt, M., & KARCI, A., “GloVe Kelime Gömmeleri ve Sinir Ağları ile Haber Metinlerinin Sınıflandırılması.”, International Journal of Pure and Applied Sciences, 9(1), (2023).
31. [31] Parlak, B., “The effects of preprocessing on Turkish and English News Data.”, Sakarya University Journal of Computer and Information Sciences, 6(1), 59-66, (2023).
32. [32] Okur, H. I. ve Sertbaş, A., “Pretrained neural models for turkish text classification.”, In 2021 6th International Conference on Computer Science and Engineering (UBMK) (pp. 174-179). IEEE, (2021).
33. [33] Demir, E., & Bilgin, M., “Sentiment Analysis from Turkish News Texts with BERT-Based Language Models and Machine Learning Algorithms.”, In 2023 8th International Conference on Computer Science and Engineering (UBMK) (pp. 01-04). IEEE, (2023).
34. [34] Tohma, K., Okur, H. I., Kutlu, Y., & Sertbas, A., “Sentiment Analysis in Turkish Question Answering Systems: An Application of Human-Robot Interaction.”, IEEE Access, (2023).
35. [35] Malekzadeh, M., Hajibabaee, P., Heidari, M., Zad, S., Uzuner, O., & Jones, J. H., “Review of graph neural network in text classification.”, In 2021 IEEE 12th annual ubiquitous computing, electronics & mobile communication conference (UEMCON) (pp. 0084-0091). IEEE, (2021).
36. [36] Huang, L., Ma, D., Li, S., Zhang, X., & Wang, H., “Text level graph neural network for text classification.”, arXiv preprint arXiv:1910.02356, (2019).
37. [37] Yao, L., Mao, C., & Luo, Y., “Graph convolutional networks for text classification.”, In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 7370-7377), (2019).
38. [38] Kumar, V. S., Alemran, A., Karras, D. A., Gupta, S. K., Dixit, C. K., & Haralayya, B., “Natural Language Processing using Graph Neural Network for Text Classification.”, In 2022 International Conference on Knowledge Engineering and Communication Systems (ICKES) (pp. 1-5). IEEE, (2022).
39. [39] Liu, C., Wang, X., & Xu, H., “Text Classification Using Document-Relational Graph Convolutional Networks.”, IEEE Access, 10, 123205-123211, (2022).
40. [40] Z. Chen et al., "Relational graph convolutional network for text-mining-based accident causal classification.", Applied Sciences 12.5 : 2482, (2022).
41. [41] Liu, X., Tian, J., Niu, N., Li, J., & Han, J., “Standard Text” Relational Classification Model Based on Concatenated Word Vector Attention and Feature Concatenation.”, Applied Sciences, 13(12), 7119, (2023).
42. [42] LeCun, Y., Bengio, Y., & Hinton, G., “Deep learning.”, Nature, 521(7553), 436-444, (2015).
43. [43] LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P., “Gradient-based learning applied to document recognition.” Proceedings of the IEEE, 86(11), 2278-2324, (1998).
44. [44] Hochreiter, S., & Schmidhuber, J., “Long short-term memory.”, Neural computation, 9(8), 1735-1780, (1997).
45. [45] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I., “Attention is all you need.”, Advances in neural information processing systems, 30, (2017).
46. [46] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., & Bengio, Y., “Graph attention networks.”, arXiv preprint arXiv:1710.10903, (2017).
47. [47] Kipf, T. N., & Welling, M., “Semi-supervised classification with graph convolutional networks.”, arXiv preprint arXiv:1609.02907, (2016).
48. [48] Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G., “The graph neural network model.”, IEEE transactions on neural networks, 20(1), 61-80, (2008).
49. [49] https://www.trthaber.com/tum-mansetler-sayfa-1.html, “TRT Haber Tüm Manşetler”, (2023).
50. [50] Hickman, L., Thapa, S., Tay, L., Cao, M., & Srinivasan, P., “Text preprocessing for text mining in organizational research: Review and recommendations.”, Organizational Research Methods, 25(1), 114-146, (2022).
51. [51] Aizawa, A., “An information-theoretic perspective of tf–idf measures.”, Information Processing & Management, 39(1), 45-65, (2003).
52. [52] Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., & Joulin, A., “Advances in pre-training distributed word representations.”, arXiv preprint arXiv:1712.09405, (2017).
53. [53] Pennington, J., Socher, R., & Manning, C. D., “Glove: Global vectors for word representation.”, In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543), (2014).
54. [54] Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., & Mikolov, T., “Fasttext. zip: Compressing text classification models.”, arXiv preprint arXiv:1612.03651, (2016).
55. [55] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K., “Bert: Pre-training of deep bidirectional transformers for language understanding.”, arXiv preprint arXiv:1810.04805, (2018).
56. [56] Hu, L., Zhang, M., Li, S., Shi, J., Shi, C., Yang, C., & Liu, Z. Text-graph enhanced knowledge graph representation learning. Frontiers in Artificial Intelligence, 4, 697, (2021).
57. [57] Çilden, E. K., “Stemming Turkish words using snowball.”, (2006).
58. [58] Bird, S., “NLTK: the natural language toolkit.”, In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions (pp. 69-72), (2006).
59. [59] Herman, I., Fernández, S., & Tejo, C., “SPARQL endpoint interface to python.”, URL: https://sparqlwrapper.readthedocs.io/en/latest/main.html , (2012).
60. [60] Fey, M., & Lenssen, J. E., “Fast graph representation learning with PyTorch Geometric.”, arXiv preprint arXiv:1903.02428, (2019).