Machine Learning and Deep Learning Based Classification of Profanity and Sexist Discourse in Turkish Texts

Abstract

In this study, harmful content detection in Turkish texts is addressed as a three-class text classification problem comprising Harmless, Profanity and Sexist Discourse. For this purpose, open-access Turkish datasets and data compiled within the scope of the TÜBİTAK 1001 project were combined, and after removing cross-source duplicates and conflicting duplicates, a final dataset of 12051 texts was used. In the experimental process, traditional machine learning models and Transformer-based language models were compared under the same dataset, the same class structure and a 5-fold group-based stratified cross-validation protocol. For the traditional models, in addition to word- and character-level TF-IDF representations, Word2Vec- and FastText-based representations were used to evaluate Linear SVM, Logistic Regression, Decision Tree, Random Forest, Extra Trees and XGBoost algorithms. For the Transformer-based models, BERTurk, Electra-TR, ConvBERTurk and XLM-RoBERTa were adapted to the three-class classification task. Among the traditional models, the highest performance was obtained with Logistic Regression on TF-IDF representations (82.82% macro F1-score); Linear SVM produced very close results, while FastText- and Word2Vec-based representations remained comparatively lower. The obtained results showed that Transformer-based models offer higher and more balanced performance than traditional machine learning models. Among all models, the highest success was achieved with the ConvBERTurk model, which reached a 90.93% macro F1-score, 90.93% accuracy and a 98.09% ROC-AUC value. The BERTurk model exhibited a performance very close to ConvBERTurk with a 90.28% macro F1-score. A paired t-test over the five folds shows that this difference is not statistically significant (t(4) = 2.36, P=0.078). These findings indicate that contextual language representations are effective in Turkish harmful content classification. The study contributes to the literature by addressing Turkish harmful content detection as a multi-class problem that also includes the distinction between profanity and sexist discourse, and by comparing traditional models and Transformer-based models under the same leakage-controlled experimental protocol.

Keywords

• Turkish text classification
• Harmful content detection
• Profanity detection
• Sexist discourse detection
• Machine learning
• Transformer models

Ethical Statement

Ethics committee approval was not required for this study because of there was no study on animals or humans.

Thanks

This study was supported by the Scientific and Technological Research Council of Türkiye (TÜBİTAK) under the 1001 Program, Project No. 124E056. The authors would like to thank all members of the project team for their valuable contributions to the compilation and labeling of the dataset. During the preparation of this work, the authors used Claude, an AI language model developed by Anthropic, to assist in drafting and refining sections of the manuscript. After using this tool, the authors reviewed and edited the content to ensure its accuracy and appropriateness. The authors take full responsibility for the content of the published article.

References

1. Aksoy, Ç., Demirezen, M. U., & Sağıroğlu, Ş. (2026). Hate speech detection in Turkish: An ensemble transformer-based deep learning approach. Engineering Applications of Artificial Intelligence, 164, 113147. https://doi.org/10.1016/j.engappai.2025.113147
2. Aliyeva, Ç. O., & Yağanoğlu, M. (2025). Deep learning approach to detect cyberbullying on Twitter. Multimedia Tools and Applications, 84(19), 20497–20520. https://doi.org/10.1007/s11042-024-19869-3
3. Altinel, A. B., Karatas Baydogmus, G., Sahin, S., & Gurbuz, M. Z. (2024). So-haTRed: A novel hybrid system for Turkish hate speech detection in social media with ensemble deep learning improved by BERT and clustered-graph networks. IEEE Access, 12, 86252–86270. https://doi.org/10.1109/ACCESS.2024.3415350
4. Barkhordar, E., Topçu, I. S., & Hürriyetoğlu, A. (2024, March). Team Curie at HSD-2Lang 2024: Hate speech detection in Turkish and Arabic tweets using BERT-based models. In Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events from Text (CASE 2024) (pp. 215–220).
5. Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5, 135–146. https://doi.org/10.1162/tacl_a_00051
6. Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324
7. Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and regression trees. Wadsworth International Group.
8. Büyükdemirci, K., Kucukkaya, I. E., Ölmez, E., & Toraman, C. (2024). JL-Hate: An annotated dataset for joint learning of hate speech and target detection. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (pp. 9543–9553).

9. Chen, T., & Guestrin, C. (2016, August). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785–794). https://doi.org/10.1145/2939672.2939785
10. Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020). ELECTRA: Pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations (ICLR 2020). https://openreview.net/forum?id=r1xMH1BtvB
11. Çöltekin, Ç. (2020, May). A corpus of Turkish offensive language on social media. In Proceedings of the Twelfth Language Resources and Evaluation Conference (pp. 6174–6184).
12. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020, July). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 8440–8451). https://doi.org/10.18653/v1/2020.acl-main.747
13. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018
14. Cox, D. R. (1958). The regression analysis of binary sequences (with discussion). Journal of the Royal Statistical Society: Series B (Methodological), 20(2), 215–242. https://doi.org/10.1111/j.2517-6161.1958.tb00292.x
15. Dehghan, S., & Yanıkoğlu, B. (2024, March). Evaluating ChatGPT’s ability to detect hate speech in Turkish tweets. In Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events from Text (CASE 2024) (pp. 54–59).
16. De Oliveira, A. B., Baptista, C. D. S., Firmino, A. A., & De Paiva, A. C. (2024, April). A large language model approach to detect hate speech in political discourse using multiple language corpora. In Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing (pp. 1461–1468). https://doi.org/10.1145/3605098.3635964
17. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019, June). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171–4186). https://doi.org/10.18653/v1/N19-1423
18. Esen, F. S., & Amrahov, S. E. (2024, October). TurkHSD: A hate speech detection model for Turkish text content. In 2024 Innovations in Intelligent Systems and Applications Conference (ASYU) (pp. 1–8). IEEE. https://doi.org/10.1109/ASYU62119.2024.10757058
19. Geurts, P., Ernst, D., & Wehenkel, L. (2006). Extremely randomized trees. Machine Learning, 63(1), 3–42. https://doi.org/10.1007/s10994-006-6226-1
20. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., & Mikolov, T. (2018, May). Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (pp. 3483–3487). European Language Resources Association.
21. Gümüş, E., Sarıyar, R., & Göz, F. (2024, September). Comparative analysis of transformer-based vector models for hate speech in Turkish tweets. In 2024 8th International Artificial Intelligence and Data Processing Symposium (IDAP) (pp. 1–5). IEEE. https://doi.org/10.1109/IDAP64064.2024.10710825
22. Jiang, Z. H., Yu, W., Zhou, D., Chen, Y., Feng, J., & Yan, S. (2020). ConvBERT: Improving BERT with span-based dynamic convolution. Advances in Neural Information Processing Systems, 33, 12837–12848.
23. Karayiğit, H. (2026). Leveraging attention-based deep learning models for detecting insult offenses on social media in the context of Turkish Penal Law. Neural Computing and Applications, 38, Article 6. https://doi.org/10.1007/s00521-025-11807-0.
24. Karayiğit, H., Akdağlı, A., & Acı, Ç. İ. (2024). Performance comparison of data balancing techniques on hate speech detection in Turkish. Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi, 30(5), 610–621. https://doi.org/10.5505/pajes.2023.40072
25. Kesgin, B. F., & Demirer, R. M. (2025). Offensive language detection in Turkish language by using NLP. Sakarya University Journal of Science, 29(1), 1–17. https://doi.org/10.16984/saufenbilder.1349956
26. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach [Preprint]. arXiv. https://arxiv.org/abs/1907.11692
27. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space [Preprint]. arXiv. https://arxiv.org/abs/1301.3781
28. Mut Altın, L. S., & Saggion, H. (2025, September). Investigating large language models’ (LLMs) capabilities for sexism detection on a low-resource language. In Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing—Natural Language Processing in the Generative AI Era (pp. 771–779). https://doi.org/10.26615/978-954-452-098-4-089
29. Najafi, A., & Varol, O. (2024). TurkishBERTweet: Fast and reliable large language model for social media analysis. Expert Systems with Applications, 255, 124737. https://doi.org/10.1016/j.eswa.2024.124737
30. Overfit-GM. (2023). Turkish texts for toxic language detection [Data set]. Hugging Face. Retrieved June 17, 2026, from https://huggingface.co/datasets/Overfit-GM/turkish-toxic-language
31. Singhal, K., & Bedi, J. (2024, March). Transformers at HSD-2Lang 2024: Hate speech detection in Arabic and Turkish tweets using BERT-based architectures. In Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events from Text (CASE 2024) (pp. 190–194).
32. Sparck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1), 11–21.
33. Tanyel, T., Alkurdi, B., & Ayvaz, S. (2022). Linguistic-based data augmentation approach for offensive language detection. In 2022 7th International Conference on Computer Science and Engineering (UBMK) (pp. 1-6). IEEE.
34. Toraman, C., Şahinuç, F., & Yilmaz, E. (2022a). Large-scale hate speech detection with cross-domain transfer. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (pp. 2215–2225).
35. Toraman, C., Şahinuç, F., & Yilmaz, E. (2022b). Gender hate speech Turkish — gender identity subset [Data set]. Hugging Face. https://huggingface.co/datasets/ctoraman/gender-hate-speech-turkish
36. Uludoğan, G., Yüksel, A. E., Tunçer, Ü. C., Işık, B., Korkmaz, Y., Akar, D., & Özgür, A. (2024, March). Detecting hate speech in Turkish print media: A corpus and a hybrid approach with target-oriented linguistic knowledge. In Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events from Text (CASE 2024) (pp. 205–214).
37. Yagci, U. U., Kolcak, A. E., & Iscan, E. (2024, March). ReBERT at HSD-2Lang 2024: Fine-tuning BERT with AdamW for hate speech detection in Arabic and Turkish. In Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events from Text (CASE 2024) (pp. 195–198).
38. Yakar, Ö., Büyüktanır, B., Çil, A. E., & Altınel Girgin, A. B. (2024). Türkçe nefret söylemi problemi analizinde farklı sınıflandırma algoritmalarının ve özellik seçimi yöntemlerinin performans karşılaştırması. Avrupa Bilim ve Teknoloji Dergisi, (53), 97–111.