Offensive Language Detection from Turkish Tweets with Deep and Shallow Machine Learning Methods

Öz

Hate speech is the general name for speech that expresses hatred towards a person or a group or encourages violence. These discourses have recently increased uncontrollably in digital environments. Written hate speech, especially on social media such as Twitter, has reached dangerous dimensions for both individuals and communities. In order to prevent the spread of hate speech in digital environments easily and quickly, systems that can automatically detect these speeches are needed. In our study, artificial intelligence models that can automatically detect 'offensive' speech, which is one of the most common hate speeches, are discussed. In our study, in which deep and shallow machine learning methods are used comparatively, the discourses in Turkish tweets can be divided into 2 categories as offensive or not. In the models we developed using a dataset with an imbalance of approximately 75%-25%, successful results are obtained with a rate of 0.85 on the accuracy and 0.74 on the f-score. The classification results obtained from shallow models trained using term frequency-inverse document frequency (tf-idf) vectors of tweets in the dataset and deep models trained using word embeddings are presented comparatively in this study. Experimental studies have shown that the hate speech detection model developed using Bidirectional Long Short-Term Memory (BiLSTM) technique produces more successful results than shallow methods and some other deep learning methods.

Anahtar Kelimeler

• Deep learning
• Machine learning
• Hate speech
• Offensive speech
• BiLSTM

Derin ve Sığ Makine Öğrenmesi Yöntemleri ile Türkçe Tweetlerden Saldırgan Dil Tespiti

Öz

Nefret söylemi, bir kişiye veya bir gruba yönelik nefreti ifade eden veya şiddeti teşvik eden söylemlerin genel adıdır. Bu söylemler son zamanlarda dijital ortamlarda kontrol edilemez bir şekilde artmıştır. Özellikle Twitter gibi sosyal mecralardaki yazılı nefret söylemleri hem kişiler hem de topluluklar için tehlikeli boyutlara ulaşmıştır. Nefret söyleminin dijital ortamlarda kolaylıkla ve hızlıca yayılabilmesinin önüne geçebilmek için bu söylemleri otomatik tespit edebilecek sistemlere ihtiyaç vardır. Çalışmamızda, en yaygın nefret söylemlerinden biri olan ‘saldırgan’ söylemleri otomatik olarak tespit edebilen yapay zeka modelleri ele alınmıştır. Derin ve sığ makine öğrenmesi yöntemlerinin karşılaştırmalı olarak kullanıldığı çalışmamızda, Türkçe tweetler’deki söylemler saldırgan veya değil olmak üzere 2 kategoriye ayrılabilmektedir. Yaklaşık %75-%25 dengesizliğindeki bir veri kümesini kullanarak geliştirdiğimiz modellerde, doğruluk ölçeğinde 0,85, f-skor ölçeğinde 0,74 oranında başarılı sonuçlar elde edilmiştir. Veri kümesinde bulunan tweetler’in terim frekansı-ters doküman frekansı (tf-idf) vektörleri kullanılarak eğitilen sığ modeller ile sözcük yerleştirmeleri kullanılarak eğitilen derin modellerden elde edilen sınıflandırma sonuçları karşılaştırmalı olarak bu çalışmada sunulmuştur. Yapılan deneysel çalışmalar ile Çift-Yönlü Uzun Kısa Süreli Bellek (BiLSTM) tekniği kullanılarak geliştirilen saldırgan söylem tespit modelinin, sığ yöntemlerden ve diğer bazı derin öğrenme yöntemlerinden daha başarılı sonuçlar ürettiği gösterilmiştir.

Anahtar Kelimeler

• Derin öğrenme
• Makine öğrenmesi
• Nefret söylemi
• Saldırgan söylem
• BiLSTM

Kaynakça

1. Statista, Number of social network users in selected countries in 2017 and 2022 (in millions), Statista, 2017
2. Fortuna P., Nunes S., A survey on automatic detection of hate speech in text, ACM Comput Surv, 2018, 51
3. T.D.K., Türk Dil Kurumu, Türk Tarih Kurumu Basımevi, 1954
4. Evans M., Weber A., Council of Europe Manuals - Human Rights in Culturally Diverse Societies (2 vols.), 2010
5. Burnap P., Williams M.L., Cyber hate speech on twitter: An application of machine classification and statistical modeling for policy and decision making, Policy Internet, 2015, 7
6. Sahi H., Kilic Y., Saglam R.B., Automated Detection of Hate Speech towards Woman on Twitter, In: UBMK 2018 - 3rd International Conference on Computer Science and Engineering, 2018
7. Dağaşan T., Automatic hate speech detection on social media: Turkish tweets as an example, 2019
8. Hüsünbeyi Z.M., Detecting hate speech in Turkish texts, 2020

9. MAYDA İ., DİRİ B., YILDIZ T., Türkçe Tweetler üzerinde Makine Öğrenmesi ile Nefret Söylemi Tespiti, European Journal of Science and Technology, 2021
10. Zampieri M., Nakov P., Rosenthal S., Atanasova P., Karadzhov G., Mubarak H., et al., SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020), ArXiv, 2020
11. Charitidis P., Doropoulos S., Vologiannidis S., Papastergiou I., Karakeva S., Towards countering hate speech against journalists on social media, Online Soc Netw Media, 2020, 17
12. Guellil I., Adeel A., Azouaou F., Chennoufi S., Maafi H., Hamitouche T., Detecting hate speech against politicians in Arabic community on social media, International Journal of Web Information Systems, 2020, 16
13. Pitropakis N., Kokot K., Gkatzia D., Ludwiniak R., Mylonas A., Kandias M., Monitoring Users’ Behavior: Anti-Immigration Speech Detection on Twitter, Mach Learn Knowl Extr, 2020, 2
14. Pronoza E., Panicheva P., Koltsova O., Rosso P., Detecting ethnicity-targeted hate speech in Russian social media texts, Inf Process Manag, 2021, 58
15. Jiang A., Yang X., Liu Y., Zubiaga A., SWSR: A Chinese dataset and lexicon for online sexism detection, Online Soc Netw Media, 2022, 27
16. Chiril P., Moriceau V., Benamara F., Mari A., Origgi G., Coulomb-Gully M., An annotated corpus for sexism detection in French tweets, In: LREC 2020 - 12th International Conference on Language Resources and Evaluation, Conference Proceedings, 2020
17. Parikh P., Abburi H., Badjatiya P., Krishnan R., Chhaya N., Gupta M., et al., Multi-label categorization of accounts of sexism using a neural framework, In: EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, 2019
18. Wullach T., Adler A., Minkov E., Character-level HyperNetworks for Hate Speech Detection, Expert Syst Appl, 2022, 205, 117571
19. Wu X.-K., Zhao T.-F., Lu L., Chen W.-N., Predicting the Hate: A GSTM Model based on COVID-19 Hate Speech Datasets, Inf Process Manag, 2022, 59, 102998
20. Plaza-del-Arco F.M., Molina-González M.D., Ureña-López L.A., Martín-Valdivia M.T., Comparing pre-trained language models for Spanish hate speech detection, Expert Syst Appl, 2021, 166
21. García-Díaz J.A., Jiménez-Zafra S.M., García-Cumbreras M.A., Valencia-García R., Evaluating feature combination strategies for hate-speech detection in Spanish using linguistic features and transformers, Complex & Intelligent Systems, 2022
22. Duwairi R., Hayajneh A., Quwaider M., A Deep Learning Framework for Automatic Detection of Hate Speech Embedded in Arabic Tweets, Arab J Sci Eng, 2021, 46
23. Al-Hassan A., Al-Dossari H., Detection of hate speech in Arabic tweets using deep learning, In: Multimedia Systems, 2021
24. Kalra S., Agrawal M., Sharma Y., Detection of Threat Records by Analyzing the Tweets in Urdu Language Exploring Deep Learning Transformer , In: Forum for Information Retrieval Evaluation, 2021
25. Ali R., Farooq U., Arshad U., Shahzad W., Beg M.O., Hate speech detection on Twitter using transfer learning, Comput Speech Lang, 2022, 74
26. Karayiğit H., Akdagli A., Aci Ç.İ., Homophobic and Hate Speech Detection Using Multilingual-BERT Model on Turkish Social Media, Information Technology and Control, 2022, 51, 356–375
27. Cagri T., Furkan Ş., Eyup Halit Yilmaz, Large-Scale Hate Speech Detection with Cross-Domain Transfer, ArXiv, 2022
28. Aizawa A., An information-theoretic perspective of tf–idf measures, Inf Process Manag, 2003, 39, 45–65
29. Canbay P., Sezer E.A., Detection of Stylometric Writeprint from the Turkish Texts, In: 2020 28th Signal Processing and Communications Applications Conference, SIU 2020 - Proceedings, 2020
30. Wang S., Zhou W., Jiang C., A survey of word embeddings based on deep learning, Computing, 2020, 102
31. Mikolov T., Chen K., Corrado G., Dean J., Efficient estimation of word representations in vector space, In: 1st International Conference on Learning Representations, ICLR 2013 - Workshop Track Proceedings, International Conference on Learning Representations, ICLR, 2013
32. Pennington J., Socher R., Manning C.D., GloVe: Global vectors for word representation, In: EMNLP 2014 - 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, 2014, 1532–1543
33. Bojanowski P., Grave E., Joulin A., Mikolov T., Enriching Word Vectors with Subword Information, Trans Assoc Comput Linguist, 2017, 5
34. Ekinci E., Classification of Imbalanced Offensive Dataset – Sentence Generation for Minority Class with LSTM, Sakarya University Journal of Computer and Information Sciences, 2022
35. Küçüksille E.U., Ateş N., Destek Vektör Makineleri ile Yaramaz Elektronik Postaların Filtrelenmesi , Türkiye Bilişim Vakfı Bilgisayar Bilimleri ve Mühendisliği Dergisi, 2013
36. Soygazi F., Mostafapour V., Inan E., TurkiS: A Turkish Sentiment Analyzer Using Domain-specific Automatic Labelled Dataset, International Journal of Intelligent Systems and Applications in Engineering, 2019, 7
37. Ganaie M.A., Tanveer M., Suganthan P.N., Snasel V., Oblique and rotation double random forest, Neural Networks, 2022, 153, 496–517
38. Yakowitz S., NEAREST‐NEIGHBOUR METHODS FOR TIME SERIES ANALYSIS, J Time Ser Anal, 1987, 8
39. Ekinci E., Takcı H., Alagöz S., Poet Classification Using ANN and DNN, Electronic Letters on Science and Engineering, 2022
40. Albawi S., Mohammed T.A., Al-Zawi S., Understanding of a convolutional neural network, In: Proceedings of 2017 International Conference on Engineering and Technology, ICET 2017, 2018
41. Siami-Namini S., Tavakoli N., Namin A.S., The Performance of LSTM and BiLSTM in Forecasting Time Series, In: Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019, 2019
42. Ekinci E., İlhan Omurca S., Özbay B., Comparative assessment of modeling deep learning networks for modeling ground-level ozone concentrations of pandemic lock-down period, Ecol Modell, 2021, 457
43. Graves A., Schmidhuber J., Framewise phoneme classification with bidirectional LSTM and other neural network architectures, In: Neural Networks, 2005
44. [44] Zhang X., Li R., Dai H., Liu Y., Zhou B., Wang Z., Localization of myocardial infarction with multi-lead bidirectional gated recurrent unit neural network, IEEE Access, 2019, 7
45. [45] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., ... & Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12, 2825-2830.