Development of a Hybrid Artificial Intelligence System to Reduce the Risk of False Positives in Turkish Emails

Öz

Although spam-filter accuracy has improved over the years, false positives can route legitimate corporate emails to spam folders. In this study, an integrated hybrid artificial intelligence system was developed to reduce the risk of false positives in Turkish emails. The system consists of two modules: (i) a detection module comparing Naïve Bayes, Logistic Regression, and Linear Support Vector Machine (Linear SVM) classifiers using TF-IDF features, and (ii) a text softening module based on mT5-small, retrained using the task transfer learning method. In the detection module, the Linear SVM achieved the best spam-class F1 score of 98.86% on a balanced dataset of 6,727 examples from five public sources plus additional project examples; after filtering near-duplicate records, this adjusted to approximately 97.3%. The softening module was retrained on 1,235 paired samples covering pure advertising and aggressively toned corporate patterns. False-positive reduction was evaluated on a held-out set of 118 emails: in the high-risk subset (>50% spam likelihood), the average spam score dropped from 74.5% to 41.0% (33.5 points), and 25 of 40 emails (62.5%) were reclassified as legitimate after softening. Output quality reached a BERTScore F1 of 0.7146 (95% CI [0.699, 0.730]), with brand, URL, and numerical information preserved at 100%, 91.4%, and 93.8%. The reductions are statistically significant (Wilcoxon, p < 10^-11), and a three-rater human evaluation of 30 pairs gives positive tone-improvement (3.98/5) and corporate-appropriateness (4.14/5) alongside moderate meaning preservation (2.88/5).

Anahtar Kelimeler

• Spam Detection
• False Positive Reduction
• Natural Language Processing
• Turkish Text Classification
• mT5
• Transfer Learning
• Style Transfer.

Türkçe E-Postalarda Yanlış Pozitif Riskini Azaltmak İçin Hibrit Bir Yapay Zekâ Sistemi Geliştirilmesi

Öz

Spam filtrelerinin doğruluğu yıllar içinde artmış olsa da, yanlış pozitif sonuçlar nedeniyle meşru kurumsal e-postalar spam klasörlerine yönlendirilebilmektedir. Bu çalışmada, Türkçe e-postalarda yanlış pozitif sonuç riskini azaltmak amacıyla entegre bir hibrit yapay zeka sistemi geliştirilmiştir. Sistem iki modülden oluşmaktadır: (i) TF-IDF özelliklerini kullanarak Naïve Bayes, Lojistik Regresyon ve Doğrusal Destek Vektör Makinesi (Linear SVM) sınıflandırıcılarını karşılaştıran bir algılama modülü ve (ii) görev aktarım öğrenimi yöntemi kullanılarak yeniden eğitilen, mT5-small tabanlı bir metin yumuşatma modülü. Algılama modülünde, Lineer SVM, beş kamuya açık kaynaktan elde edilen 6.727 örnekten oluşan dengeli bir veri seti ve ek proje örnekleri üzerinde spam sınıfında %98,86’lık en iyi F1 puanını elde etmiştir; neredeyse aynı olan kayıtlar filtrelendikten sonra bu oran yaklaşık %97,3’e düşmüştür. Yumuşatma modülü, tamamen reklam amaçlı ve agresif üslublu kurumsal kalıpları kapsayan 1.235 eşleştirilmiş örnek üzerinde yeniden eğitilmiştir. Yanlış pozitiflerin azaltılması, 118 e-postadan oluşan bir test kümesinde değerlendirildi: yüksek riskli alt kümede (spam olasılığı %50'nin üzerinde), ortalama spam puanı %74,5'ten %41,0'a (33,5 puan) düştü ve 40 e-postadan 25'i (%62,5) yumuşatma işleminden sonra meşru olarak yeniden sınıflandırıldı. Çıktı kalitesi, BERTScore F1 değeri 0,7146’ya (95% CI [0,699, 0,730]) ulaştı; marka, URL ve sayısal bilgiler sırasıyla %100, %91,4 ve %93,8 oranında korunmuştur. Bu azalmalar istatistiksel olarak anlamlıdır (Wilcoxon, p < 10^-11) ve 30 çift üzerinde üç değerlendirici tarafından yapılan insan değerlendirmesi, olumlu bir üslup iyileşmesi (3,98/5) ve kurumsal uygunluk (4,14/5) ile birlikte orta düzeyde anlam koruma (2,88/5) sonuçları vermektedir.

Anahtar Kelimeler

• Spam Algılama
• Yanlış Pozitiflerin Azaltılması
• Doğal Dil İşleme
• Türkçe Metin Sınıflandırma
• mT5
• Transfer Öğrenimi
• Stil Aktarımı.

Kaynakça

1. [1] Salloum S, Gaber T, Vadera S, Shaalan K. “A Systematic Literature Review on Phishing Email Detection Using Natural Language Processing Techniques”, IEEE Access 10 (2022) 65703–65727; doi: 10.1109/ACCESS.2022.3183083.
2. [2] Eryılmaz EE, Şahin DÖ, Kılıç E. “Filtering Turkish Spam Using LSTM From Deep Learning Techniques”, In: 2020 8th International Symposium on Digital Forensics and Security (ISDFS), (2020) 1–6; doi: 10.1109/ISDFS49300.2020.9116440.
3. [3] Şimşek H, Aydemir E. “Classification of Unwanted E-Mails (Spam) with Turkish Text by Different Algorithms in Weka Program”, Journal of Soft Computing and Artificial Intelligence 3(1) (2022) 1–10; doi: 10.55195/jscai.1104694.
4. [4] Güven ZA. “Türkçe E-postalarda Spam Tespiti için Makine Öğrenme Yöntemlerinin ve Dil Modellerinin Analizi”, EJOSAT 47 (2023) 1–6; doi: 10.31590/ejosat.1234079.
5. [5] Candan EN, Küçükilhan R, Eroğlu A. “Spam Mail Detection in Turkish and English Languages: A Holistic Study of AI-based Techniques including Individual, Ensemble and Hybrid Approaches”, Necmettin Erbakan University Journal of Science and Engineering 7(2) (2025) 189–205; doi: 10.47112/neufmbd.2025.85.
6. [6] Rao S, Tetreault J. “Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer”, In: Proceedings of NAACL-HLT 2018, New Orleans, LA, USA, (2018) 129–140; doi: 10.18653/v1/N18-1012.
7. [7] Cheng Y, Gan Z, Zhang Y, Elachqar O, Li D, Liu J. “Contextual Text Style Transfer”, In: Findings of the Association for Computational Linguistics: EMNLP 2020, (2020) 2915–2924; doi: 10.18653/v1/2020.findings-emnlp.263.
8. [8] Yao Z, Yu H. “Improving Formality Style Transfer with Context-Aware Rule Injection”, In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), (2021) 1561–1570; doi: 10.18653/v1/2021.acl-long.124.

9. [9] Dedeturk BK, Akay B. “Spam filtering using a logistic regression model trained by an artificial bee colony algorithm”, Applied Soft Computing 91 (2020) 106229; doi: 10.1016/j.asoc.2020.106229.
10. [10] Karasoy O, Ballı S. “Spam SMS Detection for Turkish Language with Deep Text Analysis and Deep Learning Methods”, Arabian Journal for Science and Engineering 47(8) (2022) 9361–9377; doi: 10.1007/s13369-021-06187-1.
11. [11] Sudhakar A, Upadhyay B, Maheswaran A. “‘Transforming’ Delete, Retrieve, Generate Approach for Controlled Text Style Transfer”, In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, (2019) 3269–3279; doi: 10.18653/v1/D19-1322.
12. [12] Turker M, Ari ME, Han A. “VBART: The Turkish LLM”, (2024); doi: 10.48550/arXiv.2403.01308.
13. [13] Pedregosa F et al. “Scikit-learn: Machine Learning in Python”, Journal of Machine Learning Research 12(85) (2011) 2825–2830.
14. [14] Xue L et al. “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer”, In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (2021) 483–498; doi: 10.18653/v1/2021.naacl-main.41.
15. [15] Vaswani A et al. “Attention Is All You Need”, In: Advances in Neural Information Processing Systems (NeurIPS) 30 (2017) 5998–6008.
16. [16] Wolf T et al. “Transformers: State-of-the-Art Natural Language Processing”, In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, (2020) 38–45; doi: 10.18653/v1/2020.emnlp-demos.6.
17. [17] Papineni K, Roukos S, Ward T, Zhu WJ. “BLEU: a method for automatic evaluation of machine translation”, In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA, (2002) 311–318; doi: 10.3115/1073083.1073135.
18. [18] Lin CY. “ROUGE: A Package for Automatic Evaluation of Summaries”, In: Text Summarization Branches Out, Barcelona, Spain, (2004) 74–81.
19. [19] Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. “BERTScore: Evaluating Text Generation with BERT”, In: International Conference on Learning Representations (ICLR), (2020).
20. [20] Briakou E, Agrawal S, Zhang K, Tetreault J, Carpuat M. “A Review of Human Evaluation for Style Transfer”, In: Proceedings of the First Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), (2021) 58–67; doi: 10.18653/v1/2021.gem-1.6.
21. [21] Uludoğan G, Balal ZY, Akkurt F, Türker M, Güngör O, Üsküdarlı S. “TURNA: A Turkish encoder-decoder language model for enhanced understanding and generation”, In: Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, (2024) 10103–10117; doi: 10.18653/v1/2024.findings-acl.600.
22. [22] de Rivero M, Tirado C, Ugarte W. “Formalstyler: GPT-based model for formal style transfer with meaning preservation”, SN Computer Science 4 (2023) 739; doi: 10.1007/s42979-023-02110-7.
23. [23] Mukherjee S, Lango M, Kasner Z, Dušek O. “A Survey of Text Style Transfer: Applications and Ethical Implications”, arXiv preprint arXiv:2407.16737, (2024); doi: 10.48550/arXiv.2407.16737.
24. [24] Lai H, Toral A, Nissim M. “Multidimensional Evaluation for Text Style Transfer Using ChatGPT”, arXiv preprint arXiv:2304.13462, (2023); doi: 10.48550/arXiv.2304.13462.
25. [25] Labonne M, Moran S. “Spam-T5: Benchmarking large language models for few-shot email spam detection”, arXiv preprint arXiv:2304.01238, (2023); doi: 10.48550/arXiv.2304.01238.
26. [26] Hotoğlu E, Şen S, Can B. “A Comprehensive Analysis of Adversarial Attacks against Spam Filters”, arXiv preprint arXiv:2505.03831, (2025); doi: 10.48550/arXiv.2505.03831.