Comparative Analysis of Traditional Machine Learning and Transformer-based Deep Learning Models for Text Classification

Year 2024, EARLY VIEW, 1 - 1

Nazif Aydın , O. Ayhan Erdem , Adem Tekerek

https://doi.org/10.2339/politeknik.1469530

Abstract

In today's information age, the generation and utilization of vast amounts of textual data have become exceedingly important. Within the realm of artificial intelligence, specifically natural language processing, text classification stands out as a crucial task, aiding in the organization and comprehension of this data deluge. The essence of text classification lies in categorizing text pieces and allocating them to respective classes, a process significantly advanced by machine learning and deep learning methodologies. The aim of this study is to evaluate the effectiveness of conventional machine learning algorithms including DT, NB, RF, and SVM alongside state-of-the-art Transformer-based models such as BERT, DistilBERT, GPT-2, GPT-3, and RoBERTa in text classification tasks. Findings indicate that while Naive Bayes achieves a 65% accuracy rate among traditional methods, GPT-3 surpasses them with a 77% higher accuracy and F1 score. These results highlight the significant promise and efficiency of Transformer-based models in text classification endeavors.

Keywords

Deep Learning, Machine Learning, Natural Language Processing, Text Classification, Transformer

Metin Sınıflandırması için Geleneksel Makine Öğrenimi ve Dönüştürücü Tabanlı Derin Öğrenme Modellerinin Karşılaştırmalı Analizi

Abstract

Günümüz bilgi çağında, büyük miktarda metinsel verinin üretilmesi ve kullanılması son derece önemli hale gelmiştir. Yapay zeka alanında, özellikle doğal dil işleme içerisinde, metin sınıflandırma bu veri selinin düzenlenmesine ve anlaşılmasına yardımcı olan kritik bir görev olarak öne çıkar. Metin sınıflandırmanın özü, metin parçalarını kategorilere ayırarak bunları ilgili sınıflara tahsis etmektir, bu süreç, makine öğrenimi ve derin öğrenme metodlarıyla önemli ölçüde ilerletilmiştir. Bu çalışmanın amacı, Geleneksel Makine Öğrenimi teknikleri arasında Karar Ağacı, Naive Bayes, Rastgele Orman ve SVM gibi tekniklerin etkinliğini değerlendirmek, ayrıca DistilBERT, BERT, GPT-2, RoBERTa ve GPT-3 gibi son teknoloji Transformer tabanlı modellerin metin sınıflandırma görevlerindeki performansını değerlendirmektir. Bulgular, Naive Bayes'in geleneksel yöntemler arasında %65'lik bir doğruluk oranına ulaştığını gösterirken, GPT-3'ün onları %77 daha yüksek bir doğruluk ve F1 skoru ile aştığını ortaya koymaktadır. Bu sonuçlar, Transformer tabanlı modellerin metin sınıflandırma çabalarında önemli vaatler ve etkinliklerini vurgulamaktadır.

Keywords

Derin Öğrenme, Makine Öğrenimi, Doğal Dil İşleme, Metin Sınıflandırma, Dönüştürücü

References

• [1] Göksu G., “Türkiye Yapay Zeka İnisiyatifi 1. Çalıştay Raporu”, GelecekHane A.Ş., (2017).
• [2] Hirvonen, N., Jylhä, V., Lao, Y., and Larsson, S., “Artificial intelligence in the information ecosystem: Affordances for everyday information seeking”, Journal of the Association for Information Science and Technology, (2023).
• [3] Aydın N. and Erdem O.A, “A research on the new generation artificial intelligence technology generative pretraining transformer 3” 2022 3rd International Informatics and Software Engineering Conference (IISEC), (2022).
• [4] Chen, H., Zhang, H., Yang, Y., and He, L.,“A text classification network model combining machine learning and deep learning”, International Journal of Sensor Networks, 44(3): 182-192, (2024).
• [5] Mokhamed, T., Harous, S., Hussein, N., and Ismail, H., “Comparative analysis of deep learning and machine learning algorithms for emoji prediction from arabic text”, Social Network Analysis and Mining, 14(1),: 67, (2024).
• [6] Chhabra, A., Arora, M., Sharma, A., Singh, H., Verma, S., Jain, R., ... and Kanavos, A., “Classifying hindi news using various machine learning and deep learning techniques”, International Journal on Artificial Intelligence Tools, (2023).
• [7] Kamath, C. N., Bukhari, S. S., and Dengel, A. “Comparative study between traditional machine learning and deep learning approaches for text classification”, In Proceedings of the ACM Symposium on Document Engineering 2018, 1-11,(2018).
• [8] Li, Q., Peng, H., Li, J., Xia, C., Yang, R., Sun, L., ... and He, L., “A survey on text classification: from traditional to deep learning.” ACM Transactions on Intelligent Systems and Technology (TIST), 13(2): 1-41, (2022).
• [9] Ikonomakis, M., Kotsiantis, S., and Tampakas, V., “Text classification using machine learning techniques“, WSEAS Transactions on Computers, 4(8): 966-974, (2005).
• [10] Mukhamediev, R. I., Popova, Y., Kuchin, Y., Zaitseva, E., Kalimoldayev, A., Symagulov, A., ... and Yelis, M., “Review of artificial intelligence and machine learning technologies: classification, restrictions, opportunities and challenges”, Mathematics, 10(15):2552, (2022).
• [11] Doszkocs, T. E., “Natural language processing in information retrieval”, Journal of the American Society for Information Science, 37(4): 191-196, (1986).
• [12] Kuşçu, E., “Çeviride Yapay Zeka Uygulamalari”, Atatürk Üniversitesi Kazım Karabekir Eğitim Fakültesi Dergisi, 30: 45-58, (2015).
• [13] Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., and Wang, Z., “ChatGPT and a new academic reality: artificial intelligence‐written research papers and the ethics of the large language models in scholarly publishing”, Journal of the Association for Information Science and Technology, 74(5): 570-581, (2023).
• [14] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... and Polosukhin, I., “Attention is all you need”, Advances in Neural Information Processing Systems, (2017).
• [15] Wang, Z., He, S., Xu, G., and Ren, M., “Will sentiment analysis need subculture? A new data augmentation approach”, Journal of the Association for Information Science and Technology, 75(6): 655-670, (2024).
• [16] Aydın, N., “A Review On The New Generation Technology Artificial Intelligence Generative Pre-Trained Transformer” Master's Thesis, Gazi University, (2023).
• [17] Amatriain, X., Sankar, A., Bing, J., Bodigutla, P. K., Hazen, T. J., and Kazi, M., “Transformer models: an introduction and catalog”, arXiv preprint arXiv:2302.07730., (2023).
• [18] Devlin, J., Chang, M. W., Lee, K., and Toutanova, K., “Bert: pre-training of deep bidirectional transformers for language understanding”, arXiv preprint arXiv:1810.04805., (2018).
• [19] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I., “Language models are unsupervised multitask learners”, OpenAI blog, 1(8):9, (2019).
• [20] Sanh, V., Debut, L., Chaumond, J., and Wolf, T., “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”, arXiv preprint arXiv:1910.01108., (2019).
• [21] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... and Stoyanov, V., “Roberta: A robustly optimized bert pretraining approach”, arXiv preprint arXiv:1907.11692., (2019).
• [22] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... and Amodei, D., “Language models are few-shot learners”, Advances in Neural Information Processing Systems, 33: 1877-1901, (2020).
• [23] Suissa, O., Elmalech, A., and Zhitomirsky‐Geffet, M., “Text analysis using deep neural networks in digital humanities and information science”, Journal of the Association for Information Science and Technology, 73(2): 268-287, (2022).
• [24] Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., and Brown, D., “Text classification algorithms: A survey”, Information, 10(4): 150, (2019).
• [25] Breiman, L., “Random forests”, Machine Learning, 45: 5-32, (2001).