Review of the Methods Used for the Detection of Spam Email

Abstract

In this study, the methods in the literature for filtering spam emails were reviewed. Emails are actively used by people or communities who want to make propaganda, advertising, phishing because of their ease of use and low cost. People or communities who want to achieve their goals send unnecessary and unsolicited mail to the email accounts they never knew. These mails cause serious financial and moral damages to internet users and also engage in internet traffic. Unsolicited emails are a method that is sent to the recipient without his consent and that is generally used by malicious or promotional purposes. In this article, important developments in spam filtering methods are evaluated and deficiencies are revealed. The filtering of spam emails has been reviewed under two main headings: non-artificial intelligence-based and artificial intelligence-based. It has been observed that non-artificial intelligence-based methods give effective results in detecting spam, but there is spam that can easily skip these methods. It has been revealed that systems based on artificial intelligence are frequently used in spam detection and research and development are in this direction. In recent years, with the development of artificial intelligence, machine learning and algorithms emerging in its deep learning field, which is a sub-branch of it, have been detected with high performance and spam email detection. Due to the high performance of machine learning and deep learning methods for filtering spam emails, studies in this field are concentrated in detecting and filtering spam email.

Keywords

• spam email detection
• artificial intelligence
• machine learning

İstenmeyen Epostaların Tespiti için Kullanılan Yöntemlerin İncelenmesi

Abstract

İstenmeyen elektronik postalar alıcıya rızası dışında gönderilen ve genellikle kötü niyetli veya tanıtım amaçlı olan kişilerin başvurduğu bir yöntemdir. Elektronik postalar, kullanımının kolaylığı, maliyetlerinin ucuz olmasından dolayı propaganda, reklam, oltalama yapmak isteyen kişi veya topluluklar tarafından etkin bir biçimde kullanılmaktadır. Amaçlarını gerçekleştirmek isteyen kişi veya topluluklar hiç tanımadıkları e-posta hesaplarına gereksiz ve istenmeyen postalar gönderirler. Bu çalışmada, istenmeyen elektronik postaların filtrelenmesi için literatürde bulunan yöntemler incelenmiştir. Bu istenmeyen e-posta filtreleme yöntemleri temel olarak yapay zekâ tabanlı olmayan ve yapay zekâ tabanlı olan şeklinde iki ana başlık altında incelenmiştir. Yapay zekâ tabanlı olmayan yöntemlerin istenmeyen e-posta tespitinde etkili sonuçlar verdiği ancak literatürde bu yöntemleri atlayabilen tekniklerin olduğu görülmektedir. İstenmeyen e-posta tespitinde yapay zekâ tabanlı makine öğrenmesi algoritmaları kullanan sistemlerin popülaritesinin arttığı ve araştırmaların bu yönde ivme kazandığı görülmektedir. Özellikle derin öğrenme yöntemleri yüksek performansları nedeniyle spam tespitinde tercih edilmeye başlamıştır. Literatürde klasik makine öğrenme yöntemlerinden olan Bayes, Destek Vektör Makinesi, Yapay Sinir Ağı, Rastgele Orman, Çok Katmanlı Algılayıcı, K-En Yakın Komşu gibi algoritmalarının kullanıldığı spam tespit yöntemlerinde yüksek başarım sağladığı görülmektedir. Uzun Kısa Süreli Bellek ve Evrişimsel Sinir Ağı algoritmalarını kullanan derin öğrenme temelli spam tespit yöntemlerinin başarım oranlarını daha da artırdığı farklı veri kümeleri kullanılarak gösterilmiştir. Ayrıca spam tespit sistemlerinde bulunan açık problemler ve Türkçe özelinde bu çalışmaların hangi aşamada olduğu da bu çalışmada irdelenmiştir ve çeşitli öneriler yapılmıştır.

Keywords

• İstenmeyen eposta tespiti
• yapay zeka
• makine öğrenmesi
• derin öğrenme

References

1. I. Peter, "The history of email", 2004. URL: http://www.nethistory.info/History%20of%20the%20Internet/email.html. (Erişim zamanı: 14- Mar- 2020).
2. BTK, “Türkiye elektronik haberleşme sektörü üç aylık pazar verileri raporu,” btk.gov.tr. URL: https://www.btk.gov.tr/uploads/pages/pazar-verileri/3-ceyrekraporu-2019.pdf. [Erişim zamanı: 14-Mar-2020].
3. M. Rathi and V. Pareek, “Spam Mail Detection through Data Mining – A Comparative Performance Analysis,” International Journal of Modern Education and Computer Science, vol. 5, no. 12, pp. 31–39, 2013.
4. Paswan, M. K., Bala, P. S., & Aghila, G. 2012, March. Spam filtering: Comparative analysis of filtering techniques. In IEEE-International Conference On Advances In Engineering, Science And Management (ICAESM-2012) (pp. 170-176). IEEE.
5. Avira. 2019. What is email spam?. https://www.avira.com/en/support-what-is-email-spam. (Erişim Tarihi: 22.02.2020).
6. Statista. 2020. Number of e-mail users worldwide from 2017 to 2023. https://www.statista.com/statistics/255080/number-of-e-mail-users-worldwide/ (Erişim Tarihi: 22.02.2020).
7. Campaignmonitor. 2019. The Shocking Truth about How Many Emails Are Sent. https://www.campaignmonitor.com/blog/email-marketing/2019/05/shocking-truth-about-how-many-emails-sent/ (Erişim Tarihi: 22.02.2020).
8. Bauer, E. 2018. 15 Outrageous Email Spam Statistics that Still Ring True in 2018. https://www.propellercrm.com/blog/email-spam-statistics. (Erişim Tarihi: 22.02.2020).

9. Karim, A., Azam, S., Shanmugam, B., Kannoorpatti, K., & Alazab, M. 2019. A Comprehensive Survey for Intelligent Spam Email Detection. IEEE Access, Access, IEEE, 7, 168261–168295. https://doi.org/10.1109/ACCESS.2019.2954791
10. Statista. 2019. Global e-mail spam rate from 2012 to 2018. https://www.statista.com/statistics/270899/global-e-mail-spam-rate/ (Erişim Tarihi: 28.02.2020).
11. Hameed, S., Kloht, T., & Fu, X. 2013. Identity based email sender authentication for spam mitigation. In Eighth International Conference on Digital Information Management (ICDIM 2013) (pp. 14-19). IEEE.
12. Ruef, M., & Young, E. 2019. Securing Email of your own Domain - SPF, DKIM and DMARC. https://doi.org/10.6084/m9.figshare.10145189
13. Konno, K. , Dan, K. , & Kitagawa, N. 2017. A Spoofed E-Mail Countermeasure Method by Scoring the Reliability of DKIM Signature Using Communication Data. Proceedings - International Computer Software and Applications Conference, 2, 43–48. https://doi.org/10.1109/COMPSAC.2017.37
14. Mastering email security with DMARC, SPF and DKIM. (cover story). (2018). Networks Asia, 1.
15. Prilepok, M., Berek, P., Platos, J., & Snasel, V. 2013. Spam detection using data compression and signatures. Cybernetics and systems, 44(6-7), 533-549.
16. Geerthik, S., & Anish, T. P. 2013. Filtering spam: Current trends and techniques. International Journal of Mechatronics, Electrical and Computer Technology Austrian E-Journals of Universal Scientific Organization, 3, 208-223.
17. Chen, J., Fontugne, R., Kato, A., & Fukuda, K. 2014. Clustering spam campaigns with fuzzy hashing. In Proceedings of the AINTEC 2014 on Asian Internet Engineering Conference (pp. 66-73).
18. Gansterer, W., Ilger, M., Lechner, P., Neumayer, R., & Strauß, J. 2005. Anti-spam methods-state of the art. Institute of Distributed and Multimedia Systems, University of Vienna, 28, 29.
19. Wang, H., Zhou, R., & Wang, Y. 2009 An anti-spam filtering system based on the naive Bayesian classifier and distributed checksum clearinghouse. In 2009 Third International Symposium on Intelligent Information Technology Application (Vol. 1, pp. 128-131). IEEE.
20. Bajaj, K. S., Egbufor, F., & Pieprzyk, J. 2011. Critical analysis of spam prevention techniques. In 2011 Third International Workshop on Security and Communication Networks (IWSCN) (pp. 83-87). IEEE.
21. Caruana, G., & Li, M. 2008. A survey of emerging approaches to spam filtering. ACM Computing Surveys (CSUR), 44(2), 1-27.
22. Liu, P., & Moh, T. S. 2016. Content based spam E-mail filtering. In 2016 International Conference on Collaboration Technologies and Systems (CTS) (pp. 218-224). IEEE.
23. Chiba, D., Akiyama, M., Yagi, T., Hato, K., Mori, T., & Goto, S. 2018. DomainChroma: Building actionable threat intelligence from malicious domain names. Computers & Security, 77, 138-161.
24. Ramachandran, A., Feamster, N., & Vempala, S. 2007. Filtering spam with behavioral blacklisting. In Proceedings of the 14th ACM conference on Computer and communications security (pp. 342-351).
25. Sirivianos, M., Kim, K., & Yang, X. 2011. Socialfilter: Introducing social trust to collaborative spam mitigation. In 2011 Proceedings IEEE INFOCOM (pp. 2300-2308). IEEE.
26. Lin, P. C., Lin, P. H., Chiou, P. R., & Liu, C. T. 2013. Detecting spamming activities by network monitoring with Bloom filters. In 2013 15th International Conference on Advanced Communications Technology (ICACT) (pp. 163-168). IEEE.
27. Revar, P., Shah, A., Patel, J., & Khanpara, P. 2017. A Review on Different types of Spam Filtering Techniques. International Journal of Advanced Research in Computer Science, 8(5).
28. Khanna, S., Chaudhry, H., & Bindra, G. S. 2012. Inbound & Outbound Email Traffic Analysis and Its SPAM Impact. In 2012 Fourth International Conference on Computational Intelligence, Communication Systems and Networks (pp. 181-186). IEEE.
29. Kumar, R. K., Poonkuzhali, G., & Sudhakar, P. 2012. Comparative study on email spam classifier using data mining techniques. In Proceedings of the International MultiConference of Engineers and Computer Scientists (Vol. 1, pp. 14-16).
30. Laorden, C., Santos, I., Sanz, B., Alvarez, G., & Bringas, P. G. 2012. Word sense disambiguation for spam filtering. Electronic Commerce Research and Applications, 11(3), 290-298.
31. Zadeh, L. A., Klir, G. J., & Yuan, B. 1996. Fuzzy sets, fuzzy logic, and fuzzy systems: selected papers (Vol. 6). World Scientific.
32. Che, H., Liu, Q., Zou, L., Yang, H., Zhou, D., & Yu, F. 2017. A content-based phishing Email detection method. In 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) (pp. 415-422). IEEE.
33. Hu, Y., Guo, C., Ngai, E. W. T., Liu, M., & Chen, S. 2010. A scalable intelligent non-content-based spam-filtering framework. Expert Systems with Applications, 37(12), 8557-8565.
34. Statista. 2019, Aralık. https://www.statista.com/statistics/263086/countries-of-origin-of-spam/. (Erişim Tarihi: 29.02.2020).
35. Bradbury, D. 2014. Can we make email secure?. Network Security, 2014(3), 13-16.
36. Nakamoto, S. 2019. Bitcoin: A peer-to-peer electronic cash system. Manubot.
37. Darwish, A. 2018. Bio-inspired computing: Algorithms review, deep analysis, and the scope of applications. Future Computing and Informatics Journal, 3(2), 231-246.
38. Greenstadt, R., & Kaminsky, M. 2002. Evolving Spam Filters Using Genetic Algorithms. Technical Report 3836. Massachusetts Institute of Technology.
39. Conrad, E. 2007. Detecting spam with genetic regular expressions. In SANS Institute Reading Room. http://www. giac. org/certified_professionals/practicals/GCIA/0 0793. php. (Erişim Tarihi: 28.02.2020).
40. Ruano-Ordás, D., Fdez-Riverola, F., & Méndez, J. R. 2018. Using evolutionary computation for discovering spam patterns from e-mail samples. Information Processing & Management, 54(2), 303-317.
41. Saleh, A. J., Karim, A., Shanmugam, B., Azam, S., Kannoorpatti, K., Jonkman, M., & Boer, F. D. 2019. An intelligent spam detection model based on artificial immune system. Information, 10(6), 209.
42. Idris, I., Selamat, A., & Omatu, S. 2014. Hybrid email spam detection model with negative selection algorithm and differential evolution. Engineering Applications of Artificial Intelligence, 28, 97-110.
43. Idris, I., Selamat, A., Nguyen, N. T., Omatu, S., Krejcar, O., Kuca, K., & Penhaker, M. 2015. A combined negative selection algorithm–particle swarm optimization for an email spam detection system. Engineering Applications of Artificial Intelligence, 39, 33-44.
44. Goodfellow, I., Bengio, Y. and Courville, A. 2016. Deep learning. The MIT Press, 775, Cambridge, Massachusetts.
45. Bhowmick, A. and Hazarika, S. M. 2016. Machine learning for E-mail spam filtering: review, techniques and trends. arXiv preprint arXiv:1606.01042.
46. LeCun, Y., Bengio, Y. and Hinton, G. 2015. Deep learning. Nature, 521:7553, 436-444.
47. Chao, W. L. 2011. Machine learning tutorial. Digital Image and Signal Processing.
48. Shrivastava, J. N., & Bindu, M. H. 2013. E-mail classification using genetic algorithm with heuristic fitness function. International Journal of Computer Trends and Technology (IJCTT), 4(8), 2956-2961.
49. Choudhary, M., & Dhaka, V. S. 2013. Automatic E-mails classification using genetic algorithm. In Special Conference Issue: National Conference on Cloud Computing and Big Data (pp. 42-49).
50. Ergin, S., Sora Gunal, E., Yigit, H., & Aydin, R. 2012. Turkish anti-spam filtering using binary and probabilistic models. Global Journal on Technology, 1.
51. Zhao, W., & Zhang, Z. 2005. An email classification model based on rough set theory. In Proceedings of the 2005 International Conference on Active Media Technology, 2005.(AMT 2005). (pp. 403-408). IEEE.
52. Altunyaprak, C. 2006. Bayes yöntemi kullanarak istenmeyen elektronik postaların filtrelenmesi. Yüksek Lisans Tezi, Muğla Üniversitesi Fen Bilimleri Enstitüsü
53. Norte Sosa, J. 2010. Spam Classification Using Machine Learning Techniques-Sinespam (Master's thesis, Universitat Politècnica de Catalunya).
54. Yumak, B. 2011. Elektronik postaların ayrıştırılmasında Naïve bayesian ve Bulanık Mantık yöntemlerinin karşılaştırılması. Yüksek Lisans Tezi, Gazi Üniversitesi Bilişim Enstitüsü, 97, Ankara.
55. Awad, W. A., & ELseuofi, S. M. 2011. Machine learning methods for spam e-mail classification. International Journal of Computer Science & Information Technology (IJCSIT), 3(1), 173-184.
56. Idris, I., & Abdulhamid, S. M. 2014. An improved AIS based e-mail classification technique for spam detection. arXiv preprint arXiv:1402.1242.
57. Bhagyashri, G., Pratap, H., & Patil, D. Y. 2013. Auto E-mails classification using bayesian filter. International Journal of Advanced technology & Engineering Research, 3(4).
58. Ateş, N. 2014. Destek vektör makineleri ve Gauss karışım modeli ile istenmeyen e-postaların tespiti. Yüksek Lisans Tezi, Süleyman Demirel Üniversitesi Fen Bilimleri Enstitüsü, 57, Samsun.
59. Dhanaraj, K. R., & Palaniswami, V. 2014. Firefly and Bayes classifier for email spam classification in a distributed environment. Aust. J. Basic Appl. Sci., 8(17), 118-130.
60. Sharma, A. K., Prajapat, S. K., & Aslam, M. 2014. A comparative study between naïve Bayes and neural network (MLP) classifier for spam email detection. Int. J. Comput. Appl.
61. Karthika, R., & Visalakshi, P. 2015. A hybrid ACO based feature selection method for email spam classification. WSEAS Trans. Comput., 14, 171-177.
62. Kumar, S., & Arumugam, S. 2015. A probabilistic neural network based classification of spam mails using particle swarm optimization feature selection. Middle-East Journal of Scientific Research, 23(5), 874-879.
63. Renuka, D. K., Visalakshi, P., & Sankar, T. 2015. Improving E-mail spam classification using ant colony optimization algorithm. Int. J. Comput. Appl, 22-26.
64. Bajaj, K. S. 2016. A Multi-layer Model to Detect Spam Email at Client Side. International Conference on Security and Privacy in Communication Systems, Springer, 334-349.
65. Tuteja, S. K. and Bogiri, N. 2016. Email Spam filtering using BPNN classification algorithm. 2016 International Conference on Automatic Control and Dynamic Optimization Techniques (ICACDOT), IEEE, 915-919.
66. Palanisamy, C., Kumaresan, T., & Varalakshmi, S. E. (2016). Combined techniques for detecting email spam using negative selection and particle swarm optimization. Int. J. Adv. Res. Trends Eng. Technol., 3.
67. Zavvar, M., Rezaei, M., & Garavand, S. 2016. Email spam detection using combination of particle swarm optimization and artificial neural network and support vector machine. International Journal of Modern Education and Computer Science, 8(7), 68.
68. Foqaha, M. A. M. 2016. Email spam classification using hybrid approach of RBF neural network and particle swarm optimization. International Journal of Network Security & Its Applications, 8(4), 17-28.
69. Sharma, A., & Suryawanshi, A. 2016. A novel method for detecting spam email using KNN classification with spearman correlation as distance measure. International Journal of Computer Applications, 136(6), 28-35.
70. Alkaht, I. J., & Al-Khatib, B. 2016. Filtering SPAM Using Several Stages Neural Networks. Int. Rev. Comp. Softw., 11, 2.
71. Rajamohana, S. P., Umamaheswari, K., & Abirami, B. 2017. Adaptive binary flower pollination algorithm for feature selection in review spam detection. In 2017 International Conference on Innovations in Green Energy and Healthcare Technologies (IGEHT) (pp. 1-4). IEEE.
72. Akinyelu, A. A., & Adewumi, A. O. 2014. Classification of phishing email using random forest machine learning technique. Journal of Applied Mathematics, 2014.
73. Yıldız, A. 2017. Kurumsal e-posta sınıflandırma sistemi. Yüksek Lisans Tezi, Gazi Üniversitesi Fen Bilimleri Enstitüsü, 82, Ankara.
74. Şahin, E. 2018. Makine öğrenme yöntemleri ve kelime kümesi tekniği ile istenmeyen e-posta / e-posta sınıflaması. Yüksek Lisans Tezi, Hacettepe Üniversitesi Fen Bilimleri Enstitüsü, 60, Ankara.
75. Kale, B. 2018. Veri madenciliği sınıflandırma algoritmaları ile e-posta önemliliğinin belirlenmesi. Yüksek Lisans Tezi, Çukuruova Üniversitesi Fen Bilimleri Enstitüsü, 120, Adana.
76. Nazlı, N. 2018. Analysis of machine learning – based spam filtering techniques. Yüksek Lisans Tezi, Çankaya University The Graduate School of Natural and Applied Sciences, 79, Ankara.
77. Al-Azzawi, F. 2018. Wrapper feature selection approach for spam e-mail filtering. Master Thesis, Erciyes University Graduate school of natural and applied science, Kayseri.
78. Salihi, A. K. A. 2019. Spam detection by using word-vector learning algorithm in online social networks. Master Thesis, Fırat University Graduate school of natural and applied sciences institute, 46, Elazığ.
79. Bousquet, O., Bottou, L., Platt, J., Koller, D., Singer, Y. and Roweis, S. 2007. Advances in neural information processing systems.
80. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S. and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 3111-3119.
81. Hochreiter, S. and Schmidhuber, J. 1997. Long Short-Term Memory. Neural Computation, 9:8, 1735-1780.
82. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A. and Bernstein, M. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:3, 211-252.
83. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A. 2015. Going deeper with convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition, 1-9.
84. Shang, E.-X. and Zhang, H.-G. 2016. Image spam classification based on convolutional neural network. 2016 International Conference on Machine Learning and Cybernetics (ICMLC), IEEE, 398-403.
85. Roy, S. S., Sinha, A., Roy, R., Barna, C. and Samui, P. 2016. Spam Email Detection Using Deep Support Vector Machine, Support Vector Machine and Artificial Neural Network. International Workshop Soft Computing Applications, Springer, 162-174.
86. Alurkar, A. A., Ranade, S. B., Joshi, S. V., Ranade, S. S., Sonewar, P. A., Mahalle, P. N. and Deshpande, A. V. 2017. A proposed data science approach for email spam classification using machine learning techniques. 2017 Internet of Things Business Models, Users, and Networks, IEEE, 1-5.
87. Seth, S. and Biswas, S. 2017. Multimodal Spam Classification Using Deep Learning Techniques. 2017 13th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), IEEE, 346-349.
88. Ertam, F. and Aydın, G. 2017. Data classification with deep learning using Tensorflow. 2017 International Conference on Computer Science and Engineering (UBMK), IEEE, 755-758.
89. Ra, V., HBa, B. G., Ma, A. K., KPa, S., Poornachandran, P. and Verma, A. 2018. DeepAnti-PhishNet: Applying deep neural networks for phishing email detection. Proc. 1st AntiPhishing Shared Pilot 4th ACM Int. Workshop Secur. Privacy Anal.(IWSPA), Tempe, AZ, USA, 1-11.
90. Yawen, W., Fan, Y. and Yanxi, W. 2018. Research of Email Classification based on Deep Neural Network. 2018 Second International Conference of Sensor Network and Computer Engineering (ICSNCE 2018), Atlantis Press.
91. Yang, H., Liu, Q., Zhou, S. and Luo, Y. 2019. A Spam Filtering Method Based on Multi-Modal Fusion. Applied Sciences, 9:6, 1152.
92. Jain, G., Sharma, M. and Agarwal, B. 2019. Optimizing semantic LSTM for spam detection. International Journal of Information Technology, 11:2, 239-250.
93. Bagui, S., Nandi, D., Bagui, S. and White, R. J. 2019. Classifying Phishing Email Using Machine Learning and Deep Learning. 2019 International Conference on Cyber Security and Protection of Digital Services (Cyber Security), IEEE, 1-2.
94. Nagisetty, A. and Gupta, G. P. 2019. Framework for Detection of Malicious Activities in IoT Networks using Keras Deep Learning Library. 2019 3rd International Conference on Computing Methodologies and Communication (ICCMC), IEEE, 633-637.
95. Roy, P. K., Singh, J. P. and Banerjee, S. 2020. Deep learning to filter SMS Spam. Future Generation Computer Systems, 102, 524-533.
96. Tyagi, A. 2016. Content Based Spam Classification-A Deep Learning Approach (Master's thesis, Graduate Studies).
97. Kingma, D. P., Mohamed, S., Rezende, D. J. and Welling, M. 2014. Semi-supervised learning with deep generative models. Advances in neural information processing systems, 3581-3589.
98. Google, URL: support.google.com (Erişim zamanı: Mart, 26, 2020).
99. Microsoft, URL: answers.microsoft.com (Erişim zamanı: Mart, 26, 2020)
100. Ott Myle, Choi Yejin, Cardie Claire, T. Hancock Jeffrey, "Finding deceptive opinion spam by any stretch of imagination", ACM Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 309-319, 2011.