Classification of Spam Content in Turkish Short Messages Using Transformer-Based Models

Öz

Short Message Service is one of the most widely used communication tools today. This widespread use has also led to an increase in the number of unwanted messages. These messages, sent for the purposes of fraud, advertising, and promotion, are messages that users do not wish to receive. This study addresses the classification of messages received via SMS into spam and non-spam categories. To this end, a project was developed using a specialized SMS dataset. Preprocessing steps were applied to the dataset to remove meaningless features from the text, and then two different classical machine learning algorithms and two different deep learning algorithms were used. These algorithms are Naive Bayes, Random Forest, BERTurk, and Turkish ELECTRA, respectively. Classification models were developed using these algorithms. During the training process, the dataset was split into an 80% training set and a 20% test set; additionally, a 5-fold cross-validation method was applied to verify the stability of the results. The models were trained on the preprocessed data and analyzed and compared using performance metrics such as Precision, Recall, F1-Score, and Accuracy. Analyses conducted on the dataset indicate that the unprocessed BERTurk model, evaluated using 5-fold cross-validation, achieved the best result with an accuracy of 0.990. The results demonstrate that different algorithms offer distinct advantages depending on the data structure.

Anahtar Kelimeler

• Short message service
• Spam detection
• Machine learning
• Deep learning
• Data preprocessing

Etik Beyan

The manuscript does not require approval from an ethics committee. There is no conflict of interest with any individual or institution regarding the manuscript.

Türkçe Kısa Mesajlarda Spam İçeriğinin Transformer Tabanlı Modeller ile Sınıflandırılması

Öz

Kısa Mesaj Servisi, günümüzde en yaygın kullanılan iletişim araçlarından biridir. Bu yaygın iletişim aynı zamanda istenmeyen mesajların sayısını da artırmaktadır. Dolandırıcılık, reklam ve tanıtım amacıyla gönderilen bu mesajlar, kullanıcıların görmek istemediği mesajlardır. Bu çalışma, kısa mesaj servisi aracılığıyla alınan mesajların spam veya spam olmayan olarak sınıflandırılmasını ele almaktadır. Bu amaçla, özel bir SMS veri seti kullanılarak proje oluşturulmuştur. Veri setine ön işleme adımları uygulanarak metinden anlamsız özellikler çıkarılmış ve ardından iki farklı klasik makine öğrenimi algoritması ve iki farklı derin öğrenme algoritması kullanılmıştır. Bu algoritmalar sırasıyla Naive Bayes, Random Forest, BERTurk ve Türkçe ELECTRA'dır. Bu algoritmalar kullanılarak sınıflandırma modelleri geliştirilmiştir. Modellerin eğitim sürecinde veri seti %80 eğitim ve %20 test olacak şekilde ayrılmış, ayrıca sonuçların kararlılığını doğrulamak amacıyla 5-katlı çapraz doğrulama yöntemi uygulanmıştır. Modeller, ön işleme tabi tutulmuş veriler üzerinde eğitilmiş ve Precision, Recall, F1-Score ve Accuracy gibi performans metrikleri kullanılarak analiz edilmiş ve karşılaştırılmıştır. Veri seti üzerinde yapılan analizler, 5-kat çapraz doğrulamalı ön işleme yapılmamış BERTurk'ün modelinin 0,990 doğruluk değeri ile en iyi sonucu elde ettiğini göstermektedir. Sonuçlar, farklı algoritmaların veri yapısına bağlı olarak farklı avantajlar sunduğunu ortaya koymuştur.

Anahtar Kelimeler

• Kısa mesaj servisi
• Spam tespiti
• Makine öğrenmesi
• Derin öğrenme
• Veri ön işleme

Etik Beyan

Makalenin etik kurul onayına ihtiyacı yoktur. Makale ile ilgili olarak herhangi bir kişi veya kurumla çıkar çatışması bulunmamaktadır.

Kaynakça

1. J. Brown, B. Shipman, and R. Vetter, “SMS: The short message service,” Computer, vol. 40, no. 12, pp. 106–110, 2007.
2. Türkiye İstatistik Kurumu (TÜİK), İstatistiklerle Aile, 2024, [Online]. Accessed: Jun. 27, 2025. Available: https://data.tuik.gov.tr/Bulten/Index?p=Istatistiklerle-Aile-2024-53898
3. T. A. Almeida, T. P. Silva, I. Santos, and J. M. G. Gómez Hidalgo, “Text normalization and semantic indexing to enhance instant messaging and SMS spam filtering,” Knowl.-Based Syst., vol. 108, pp. 25–32, 2016.
4. E. Kılıç, S. N. Arslan, and M. A. Güvensan, “3-Tier hybrid approach for SMS filtering,” in Proc. 22nd Signal Process. Commun. Appl. Conf. (SIU), 2014, pp. 1950–1953.
5. Türkiye Büyük Millet Meclisi (TBMM), “Elektronik Ticaretin Düzenlenmesi Hakkında Kanun (Kanun No: 6563),” T.C. Resmî Gazete, no. 29166, Nov. 5, 2014.
6. O. Karasoy and S. Ballı, “Classification Turkish SMS with deep learning tool Word2Vec,” in Proc. Int. Conf. Comput. Sci. Eng. (UBMK), 2017, pp. 294–297.
7. H. C. Altunay and Z. Albayrak, “SMS spam detection system based on deep learning architectures for Turkish and English messages,” Appl. Sci., vol. 14, no. 24, p. 11804, 2024.
8. D. A. Oyeyemi and A. K. Ojo, “SMS Spam Detection and Classification to Combat Abuse in Telephone Networks Using Natural Language Processing,” arXiv preprint arXiv:2406.06578, 2024.

9. M. A. Bayram, A. A. Fincan, A. S. Gümüş, S. Karakaş, B. Diri, and S. Yıldırım, “Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark,” arXiv preprint arXiv:2502.07057, 2025.
10. T. A. Almeida, J. M. G. Hidalgo, and A. Yamakami, “Contributions to the study of SMS spam filtering: new collection and results,” in Proc. 11th ACM Symp. Doc. Eng., 2011, pp. 259–262.
11. E. Ezpeleta, U. Zurutuza, and J. M. G. Hidalgo, “Short messages spam filtering using personality recognition,” in Proc. 4th Spanish Conf. Inf. Retrieval, 2016, pp. 1–7.
12. O. Abayomi-Alli, S. Misra, A. Abayomi-Alli, and M. Odusami, “A review of soft techniques for SMS spam classification: Methods, approaches and applications,” Eng. Appl. Artif. Intell., vol. 86, pp. 197–212, 2019.
13. V. Goswami, V. Malviya, and P. Sharma, “Detecting spam emails/SMS using Naive Bayes, support vector machine and Random Forest,” in Proc. Int. Conf. Comput. Netw., Big Data IoT (ICCBI), 2020, pp. 306–313.
14. L. GuangJun, S. Nazir, H. U. Khan, and A. U. Haq, “Spam detection approach for secure mobile message communication using machine learning algorithms,” Secur. Commun. Netw., vol. 2020, no. 1, Art. no. 8873639, 2020.
15. S. D. Gupta, S. Saha, and S. K. Das, “SMS spam detection using machine learning,” J. Phys. Conf. Ser., vol. 1797, no. 1, Art. no. 012017, 2021.
16. R. Dubey, S. Mishra, and D. Choubey, “Recognizing spam emails/SMS using Naive Bayes and Support Vector Machine,” Complex Syst. Complex. Sci., vol. 8, pp. 1–5, 2021.
17. A. Tekerek, “Support vector machine based spam SMS detection,” Politeknik Derg., vol. 22, no. 3, pp. 779–784, 2019.
18. Ö. Örnek, “Orange 3 ile Türkçe ve İngilizce SMS mesajlarında spam tespiti,” Eskişehir Türk Dünyası Uyg. Araşt. Merk. Bilişim Derg., vol. 1, no. 1, pp. 1–4, 2019.
19. B. Parlak, “Makine öğrenmesi algoritmaları ile Türkçe için istenmeyen SMS filtreleme,” Acad. Perspect. Procedia, vol. 5, no. 3, pp. 537–546, 2022.
20. O. Karasoy and S. Ballı, “Spam SMS detection for Turkish language with deep text analysis and deep learning methods,” Arab. J. Sci. Eng., vol. 47, no. 8, pp. 9361–9377, 2022.
21. T. Şenol and H. Takcı, “Makine öğrenmesi yöntemleriyle doğruluğu yüksek SMS spam tespiti,” in Proc. 15th Int. Conf. Inf. Commun. Res. (ICAR), Mar. 2025.
22. M. F. Johari, K. L. Chiew, A. R. Hosen, K. S. C. Yong, A. S. Khan, I. A. Abbasi, and D. Grzonka, “Key insights into recommended SMS spam detection datasets,” Sci. Rep., vol. 15, no. 1, p. 8162, 2025.
23. U. Maqsood, S. Ur Rehman, T. Ali, K. Mahmood, T. Alsaedi, and M. Kundi, “An intelligent framework based on deep learning for SMS and e-mail spam detection,” Appl. Comput. Intell. Soft Comput., vol. 2023, no. 1, Art. no. 6648970, 2023.
24. H. Yang, Q. Liu, S. Zhou, and Y. Luo, “A spam filtering method based on multi-modal fusion,” Appl. Sci., vol. 9, no. 6, p. 1152, 2019.
25. A. Ghourabi, M. A. Mahmood, and Q. M. Alzubi, “A hybrid CNN-LSTM model for SMS spam detection in Arabic and English messages,” Future Internet, vol. 12, no. 9, p. 156, 2020.
26. P. K. Roy, J. P. Singh, and S. Banerjee, “Deep learning to filter SMS spam,” Future Gener. Comput. Syst., vol. 102, pp. 524–533, 2020.
27. G. Airlangga, “A comparative analysis of deep learning models for SMS spam detection: CNN-LSTM, CNN-GRU, and ResNet approaches,” J. Comput. Netw. Archit. High Perform. Comput., vol. 6, no. 4, 2024.
28. M. R. Bishi, N. S. Manikanta, G. H. S. Bharadwaj, P. S. K. Teja, and G. R. K. Rao, “Optimizing SMS spam detection: Leveraging the strength of a voting classifier ensemble,” Int. J. Intell. Syst. Appl. Eng., 2024.
29. Ö. Şengel, “A comparative analysis of learning techniques in the context of Turkish spam detection,” Batman Üniv. Yaşam Bil. Derg., vol. 14, no. 1, pp. 43–56, 2024.
30. S. N. Ilyasa and A. O. Khadidos, “Optimized SMS spam detection using SVM-DistilBERT and voting classifier: A comparative study on the impact of lemmatization,” Int. J. Adv. Comput. Sci. Appl., vol. 15, no. 11, 2024.
31. H. Al-Kaabi, F. Al-Ibraheemi, A. K. Jasim, and A. R. Mohammed, “Fusion-Based Hybrid Model for SMS Spam Detection Integrating Local, Sequential, and Contextual Features,” Res. Square, 2025.
32. Y. Bilgen and M. Kaya, “EGMA: Ensemble learning-based hybrid model approach for spam detection,” Appl. Sci., vol. 14, no. 21, p. 9669, 2024.
33. E. Cambria and B. White, “Jumping NLP curves: A review of natural language processing research,” IEEE Comput. Intell. Mag., vol. 9, no. 2, pp. 48–57, 2014.
34. D. Jurafsky and J. H. Martin, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Upper Saddle River, NJ, USA: Prentice Hall, 2009.
35. J. Ramos et al., “Using tf-idf to determine word relevance in document queries,” in Proc. 1st Instr. Conf. Mach. Learn., vol. 242, no. 1, pp. 29–48, 2003.
36. S. J. Russell and P. Norvig, Instructor's Solution Manual for Artificial Intelligence: A Modern Approach. Pearson Education Company, 2003.
37. J. D. Rennie, L. Shih, J. Teevan, and D. R. Karger, “Tackling the poor assumptions of naive Bayes text classifiers,” in Proc. 20th Int. Conf. Mach. Learn. (ICML), 2003, pp. 616–623.
38. S. Zhai, R. Li, Y. Yang, and X. Zhao, “Chain store placement recommendation with multi-source urban data,” in Proc. 6th Int. Conf. Intell. Comput. Signal Process. (ICSP), 2021, pp. 203–207.
39. L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, 2001.
40. S. Schweter, BERTurk - BERT Models for Turkish. Zenodo, 2020.
41. B. Yıldırım, R. E. Gökkaya, M. Arzu, M. E. Kaya, and M. E. Çolak, “Clickbait detection in Turkish news headlines using machine learning methods,” in Proc. 18th Int. Conf. Inf. Secur. Cryptol. (ISCTürkiye), 2025, pp. 1–6.
42. Y. Şimşek, M. B. Balci, M. Arzu, M. Kaya, and Y. Santur, “Multi-class sentiment analysis with e-commerce user reviews: Comparisons of classical and deep learning applications,” in Proc. 9th Int. Artif. Intell. Data Process. Symp. (IDAP), 2025, pp. 1–6.
43. M. Arzu and M. Aydoğan, “Türkçe duygu sınıflandırma için transformers tabanlı mimarilerin karşılaştırılmalı analizi,” Comput. Sci., IDAP-2023, pp. 1–6, 2023.
44. M. Şeker, M. A. Demir, M. Arzu, and M. Kaya, “Named entity recognition for historical texts in Turkish,” in Proc. 10th Int. Conf. Comput. Sci. Eng. (UBMK), 2025, pp. 622–627.
45. M. Arzu and M. Aydoğan, “Comparison of transformer-based Turkish models for question-answering task,” Balkan J. Electr. Comput. Eng., vol. 12, no. 4, pp. 387–393, 2025.
46. dbmdz, “electra-base-turkish-cased-discriminator,” Hugging Face, [Online]. Available: https://huggingface.co/dbmdz/electra-base-turkish-cased-discriminator. Accessed: Oct. 30, 2025.
47. M. Sokolova and G. Lapalme, “A systematic analysis of performance measures for classification tasks,” Inf. Process. Manag., vol. 45, no. 4, pp. 427–437, 2009.