Sentiment Analysis in Turkish Texts: The Effect of Deep Learning Approaches and Preprocessing Processes on Model Performance

Öz

Nowadays, with the increased use of computers, a surge in data production has emerged, making data access more convenient. In this context, a substantial amount of textual data is generated on e-commerce sites, social media, and various electronic platforms. Analyzing and extracting meaningful insights from this amassed data proves valuable for numerous institutions, organizations, and individuals. Sentiment analysis is a commonly employed technique to derive sentiments from textual data, and contemporary sentiment analysis models often leverage the high performance offered by deep learning approaches. Prior to model training, several pre-processing steps are typically applied to the text data. In this study, three distinct deep learning approaches were proposed for sentiment analysis. These models were analyzed on two different datasets: winvoker and Beyazperde. Hyper-parameters and depth of models were optimized using the Bayesian optimization method to enhance the accuracy of model. Additionally, the impact of various pre-processing techniques on model performance were assessed. When non-preprocessed data is utilized, the models trained on the winvoker dataset achieve an accuracy of 94.16%, while those trained on the Beyazperde dataset reach 86.64%. With the application of pre-processing, these accuracies improve to 94.64% for the winvoker dataset and 89.08% for the Beyazperde dataset. Based on these findings, it was concluded that the effect of pre-processing decreased and the accuracy was higher for the winvoker data set with a higher number of samples.

Anahtar Kelimeler

• sentiment analysis
• deep learning
• text pre-processing
• convolutional neural networks
• transformer networks

Türkçe Metinlerde Duygu Analizi: Derin Öğrenme Yaklaşımlarının ve Ön İşlem Süreçlerinin Model Performansına Etkisi

Öz

Günümüzde bilgisayar kullanımın artması ile birlikte insanlar daha fazla veri üretmeye başlamış ve verilere ulaşım kolaylaşmıştır. Bu bağlamda e-ticaret sitelerinde, sosyal medyada ya da diğer elektronik platformlarda çok fazla metin verisi üretilmiştir. Toplanan bu verilerin analiz edilerek anlamlandırılması birçok kurum, kuruluş ya da birey için faydalı bilgiler sağlamaktadır. Bu amaç doğrultusunda duygu analizi günümüzde sıklıkla uygulanmaktadır. Duygu analizi modellerinde derin öğrenme yaklaşımları oldukça yüksek performans göstermekte ve model eğitimi yapılmadan önce metinlere birkaç ön işlem uygulanmaktadır. Bu çalışmada duygu analizi için, evrişimsel sinir ağı, Transfomer ve hibrit olmak üzere üç farklı derin öğrenme yaklaşımı önerilmiş ve modeller winvoker ve Beyazperde olmak üzere iki farklı veri seti kullanılarak analiz edilmiştir. Modellerin doğruluğunu artırmak için hiper-parametreleri ve model derinliklileri Bayesian optimizasyon yöntemi kullanılarak optimize edilmiştir. Ön işlem süreçlerinin model performansına etkisini ölçmek için veri setlerine çeşitli ön işlem yapılarak analizler tekrar edilmiştir. Ön işlem uygulanmamış veriler kullanıldığında, winvoker veri seti ile eğitilen modellerde %94,16, Beyazperde veri seti ile eğitilen modellerde ise %86,64 doğruluğa ulaşılmıştır. Ön işlem uygulandığında ise bu başarı oranları, winvoker veri seti ile eğitilen modellerde %94,64, Beyazperde veri seti ile eğitilen modellerde ise %89,08 değerlerine ulaşmıştır. Bu sonuçlar doğrultusunda örnek sayısı daha fazla olan winvoker veri seti için ön işlemlerin etkisinin azaldığı ve doğruluğun daha yüksek olduğu sonucu çıkarılmıştır.

Anahtar Kelimeler

• duygu analizi
• derin öğrenme
• metin ön işlem
• evrişimsel sinir ağları
• transformer ağları

Kaynakça

1. Hovy EH. “What are Sentiment, Affect, and Emotion? Applying the Methodology of Michael Zock to Sentiment Analysis”, Language Production, Cognition, and the Lexicon, 2015, 13-24.
2. Sánchez-Rada JF, Iglesias CA. “Social context in sentiment analysis: Formal definition, overview of current trends and framework for comparison”, Information Fusion, 2019, 52, 344-356.
3. Wankhade M, Rao ACS, Kulkarni C. “A survey on sentiment analysis methods, applications, and challenges”, Artif Intell Review, 2022, 55(7), 5731-5780.
4. Singh T, Kumari M. “Role of Text Pre-processing in Twitter Sentiment Analysis”, Procedia Computer Science, 2016, 89, 549-554.
5. Adali E, Adamov AZ. “Sentiment analysis for agglutinative languages”, IEEE 10th International Conference on Application of Information and Communication Technologies, 2016, Bakü, Azerbaycan, 1-3.
6. Yıldırım E, Çetin FS, Eryiğit G, Temel T. “The Impact of NLP on Turkish Sentiment Analysis”, Türkiye Bilişim Vakfı Bilgisayar Bilimleri ve Mühendisliği Dergisi, 2014, 7(1).
7. Singh J, Singh G, Singh R. “Optimization of sentiment analysis using machine learning classifiers”, Human-centric Computing and information Sciences, 2017, 7(1), 1-12.
8. Jagdale RS, Shirsat VS, Deshmukh SN. “Sentiment Analysis on Product Reviews Using Machine Learning Techniques”, Cognitive Informatics and Soft Computing, 2019, 639-647.

9. Jain AP, Dandannavar P. “Application of machine learning techniques to sentiment analysis”, 2nd International Conference on Applied and Theoretical Computing and Communication Technology, 2016, Karnataka, Hindistan, 628-632.
10. Gupta B, Negi M, Vishwakarma K, Rawat G, Badhani P. “Study of Twitter Sentiment Analysis using Machine Learning Algorithms on Python”, International Journal of Computer Applications, 2017, 165, 29-34.
11. Shamantha RB, Shetty SM, Rai P. “Sentiment Analysis Using Machine Learning Classifiers: Evaluation of Performance”, IEEE 4th International Conference on Computer and Communication Systems, 2019, Singapore, 21-25.
12. Naresh A, Venkata Krishna P. “An efficient approach for sentiment analysis using machine learning algorithm”, Evolutionary intelligence, 2021, 14(2), 725-731.
13. Araque O, Corcuera-Platas I, Sánchez-Rada JF, Iglesias CA. “Enhancing deep learning sentiment analysis with ensemble techniques in social applications”, Expert Systems with Applications, 2017, 77, 236-246.
14. Jianqiang Z, Xiaolin G, Xuejun Z. “Deep Convolution Neural Networks for Twitter Sentiment Analysis”, IEEE Access, 2018, 6, 23253-23260.
15. Xu G, Meng Y, Qiu X, Yu Z, Wu X. “Sentiment Analysis of Comment Texts Based on BiLSTM”, IEEE Access, 2019, 7, 51522-51532.
16. Zhao W vd. “Weakly-Supervised Deep Embedding for Product Review Sentiment Analysis”, IEEE Transactions on Knowledge and Data Engineering, 2018, 30(1), 185-197.
17. Basiri ME, Nemati S, Abdar M, Cambria E, Acharya UR. “ABCDM: An Attention-based Bidirectional CNN-RNN Deep Model for sentiment analysis”, Future Generation Computer Systems, 2021, 115, 279-294.
18. Mishev K, Gjorgjevikj A, Vodenska I, Chitkushev LT, Trajanov D. “Evaluation of Sentiment Analysis in Finance: From Lexicons to Transformers”, IEEE Access, 2020, 8, 31662-131682.
19. Naseem U, Razzak I, Musial K, Imran M. “Transformer based Deep Intelligent Contextual Embedding for Twitter sentiment analysis”, Future Generation Computer Systems, 2020, 113, 58-69.
20. Zhang T, Gong X, Chen CLP. “BMT-Net: Broad Multitask Transformer Network for Sentiment Analysis”, IEEE Transactions on Cybernetics, 2022, 52(7), 6232-6243.
21. Myagmar B, Li J, Kimura S. “Cross-Domain Sentiment Classification With Bidirectional Contextualized Transformer Language Models”, IEEE Access,2019, 7, 163219-163230.
22. Potamias RA, Siolas G, Stafylopatis AG. “A transformer-based approach to irony and sarcasm detection”, Neural Computing and Applications, 2020, 32(23), 17309-17320.
23. Murfi H, Syamsyuriani, Gowandi T, Ardaneswari G, Nurrohmah S. “BERT-based combination of convolutional and recurrent neural network for indonesian sentiment analysis”, Applied Soft Computing, 2024, 151, 111112.
24. Liu Z, Zhou B, Chu D, Sun Y, Meng L. “Modality translation-based multimodal sentiment analysis under uncertain missing modalities”, Information Fusion, 2024, 101, 101973.
25. Dehkharghani R, Saygin Y, Yanikoglu B, Oflazer K. “SentiTurkNet: a Turkish polarity lexicon for sentiment analysis”, Lang Resources & Evaluation, 2016, 50(3), 667-685.
26. Salur MU, Aydin İ. “Türkçe Tweetler için Derin Özellik Çıkarımı Tabanlı Yeni Bir Duygu Sınıflandırma Modeli”, Fırat Üniversitesi Mühendislik Bilimleri Dergisi, 2022, 34(1).
27. Ayvaz S, Yıldırım S, Salman YB. “Türkçe Duygu Kütüphanesi Geliştirme: Sosyal Medya Verileriyle Duygu Analizi Çalışması”, European Journal of Science and Technology, 2019, 16, 51-60.
28. Çetin FS, Eryiğit G. “Türkçe Hedef Tabanlı Duygu Analizi İçin Alt Görevlerin İncelenmesi – Hedef Terim, Hedef Kategori ve Duygu Sınıfı Belirleme”, Bilişim Teknolojileri Dergisi, 2018, 11(1), 43-56.
29. Onan A. “Evrişimli Sinir Ağı Mimarilerine Dayalı Türkçe Duygu Analizi”, European Journal of Science and Technology, 2020, 374-380.
30. Tuzcu S. “Çevrimiçi Kullanıcı Yorumlarının Duygu Analizi ile Sınıflandırılması”, Eskişehir Türk Dünyası Uygulama ve Araştırma Merkezi Bilişim Dergisi, 2020, 1(2).
31. “winvoker/turkish-sentiment-analysis-dataset · Datasets at Hugging Face”, 23 Ocak 2024, https://huggingface.co/datasets/winvoker/turkish-sentiment-analysis-dataset.
32. “Turkish Sentiment Analysis Data - beyazperde.com”, 23 Ocak 2024, https://www.kaggle.com/datasets/ozcan15/turkish-sentiment-analysis-data-beyazperdecom.
33. “Multimedia Information Retrieval Laboratory - HUBBM”, 21 Mart 2024, http://humirapps.cs.hacettepe.edu.tr/tsad.aspx.
34. “string — Common string operations”, Python documentation, 24 Ocak 2024, https://docs.python.org/3/library/string.html.
35. “NLTK :: Natural Language Toolkit”, 24 Ocak 2024, https://www.nltk.org/.
36. “zemberek-python: Python port of open source text processing library for Turkish, zemberek-nlp”, 24 Ocak 2024, https://www.loodos.com.tr/.
37. “Keras: Deep Learning for humans”, 24 Ocak 2024, https://keras.io/.
38. Jones DR. “A Taxonomy of Global Optimization Methods Based on Response Surfaces”, Journal of Global Optimization, 2011, 21(4), 345-383.
39. Wu J, Chen XY, Zhang H, Xiong LD, Lei H, Deng SH. “Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimizationb”, Journal of Electronic Science and Technology, 2019, 17(1), 26-40.
40. Görmez Y, Aydin Z. “IGPRED-MultiTask: A Deep Learning Model to Predict Protein Secondary Structure, Torsion Angles and Solvent Accessibility”, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2023, 20(2), 1104-1113.
41. “scikit-optimize: sequential model-based optimization in Python — scikit-optimize 0.8.1 documentation”, 25 Ocak 2024, https://scikit-optimize.github.io/stable/.
42. “Metrics and scoring: quantifying the quality of predictions”, scikit-learn, 21 Mart 2024, https://scikit-learn/stable/modules/model_evaluation.html.
43. Yildirim M, Okay FY, Ozdemir S. “Sentiment Analysis for Turkish Unstructured Data by Machine Translation”, IEEE International Conference on Big Data (Big Data), 2020, Atlanta, GA, USA, 4811-4817.
44. Alqaraleh S. “Turkish Sentiment Analysis System via Ensemble Learning”, European Journal of Science and Technology, 2020, 4(3), 122-129.
45. Acikalin UU, Bardak B, Kutlu M. “Turkish Sentiment Analysis Using BERT”, 28th Signal Processing and Communications Applications Conference (SIU), 2020, Gaziantep, Türkiye, 1-4.