Enhancing Deep Learning-Based Sentiment Analysis Using Static and Contextual Language Models

Abstract

Sentiment Analysis (SA) is an essential task of Natural Language Processing and is used in various fields such as marketing, brand reputation control, and social media monitoring. The various scores generated by users in product reviews are essential feedback sources for businesses to discover their products' positive or negative aspects. However, it takes work for businesses facing a large user population to accurately assess the consistency of the scores. Recently, automated methodologies based on Deep Learning (DL), which utilize static and especially pre-trained contextual language models, have shown successful performances in SA tasks. To address the issues mentioned above, this paper proposes Multi-layer Convolutional Neural Network-based SA approaches using Static Language Models (SLMs) such as Word2Vec and GloVe and Contextual Language Models (CLMs) such as ELMo and BERT that can evaluate product reviews with ratings. Focusing on improving model inputs by using sentence representations that can store richer features, this study applied SLMs and CLMs to the inputs of DL models and evaluated their impact on SA performance. To test the performance of the proposed approaches, experimental studies were conducted on the Amazon dataset, which is publicly available and considered a benchmark dataset by most researchers. According to the results of the experimental studies, the highest classification performance was obtained by applying the BERT CLM with 82% test and 84% training accuracy scores. The proposed approaches can be applied to various domains' SA tasks and provide insightful decision-making information.

Keywords

• Deep learning
• Natural language processing
• Sentiment analysis
• Static language models
• Contextual language models

References

1. [1] J. Hartmann, M. Heitmann, C. Siebert, and C. Schamp, “More than a feeling: Accuracy and application of sentiment analysis”, Int. J. Res. Mark., vol. 40, no. 1, pp. 75–87, 2023.
2. [2] H. T. Phan, N. T. Nguyen, and D. Hwang, “Aspect-level sentiment analysis: A survey of graph convolutional network methods”, Inf. Fusion, vol. 91, pp. 149–172, 2023.
3. [3] F. Lin, S. Liu, C. Zhang, J. Fan, and Z. Wu, “StyleBERT: Text-audio sentiment analysis with Bi-directional Style Enhancement”, Inf. Syst., vol. 114, no. 102147, p. 102147, 2023.
4. [4] M. M. Hasan and H. Jiang, “Political sentiment and corporate social responsibility”, Br. Account. Rev., vol. 55, no. 1, p. 101170, 2023.
5. [5] D. Antypas, A. Preece, and J. Camacho-Collados, “Negativity spreads faster: A large-scale multilingual Twitter analysis on the role of sentiment in political communication”, arXiv [cs.CL], 2022.
6. [6] A. R. Rahmanti et al., “Social media sentiment analysis to monitor the performance of vaccination coverage during the early phase of the national COVID-19 vaccine rollout”, Comput. Methods Programs Biomed., vol. 221, no. 106838, p. 106838, 2022.
7. [7] R. Haque, N. Islam, M. Tasneem, and A. K. Das, “Multi-class sentiment classification on Bengali social media comments using machine learning”, International Journal of Cognitive Computing in Engineering, vol. 4, pp. 21–35, 2023.
8. [8] C. Qian, N. Mathur, N. H. Zakaria, R. Arora, V. Gupta, and M. Ali, “Understanding public opinions on social media for financial sentiment analysis using AI-based techniques”, Inf. Process. Manag., vol. 59, no. 6, p. 103098, 2022.

9. [9] H.-C. K. Lin, T.-H. Wang, G.-C. Lin, S.-C. Cheng, H.-R. Chen, and Y.-M. Huang, ‘Applying sentiment analysis to automatically classify consumer comments concerning marketing 4Cs aspects’, Appl. Soft Comput., vol. 97, no. 106755, p. 106755, 2020.
10. [10] D. Sunitha, R. K. Patra, N. V. Babu, A. Suresh, and S. C. Gupta, “Twitter sentiment analysis using ensemble based deep learning model towards COVID-19 in India and European countries”, Pattern Recognit. Lett., vol. 158, pp. 164–170, 2022.
11. [11] N. Leelawat et al., “Twitter data sentiment analysis of tourism in Thailand during the COVID-19 pandemic using machine learning”, Heliyon, vol. 8, no. 10, p. e10894, 2022.
12. [12] M. Bhattacharya, S. Bhat, S. Tripathy, A. Bansal, and M. Choudhary, “Improving biomedical named entity recognition through transfer learning and asymmetric tri-training”, Procedia Comput. Sci., vol. 218, pp. 2723–2733, 2023.
13. [13] A. Goyal, V. Gupta, and M. Kumar, “A deep learning-based bilingual Hindi and Punjabi named entity recognition system using enhanced word embeddings”, Knowl. Based Syst., vol. 234, no. 107601, p. 107601, 2021.
14. [14] Q. Qiu, M. Tian, K. Ma, Y. J. Tan, L. Tao, and Z. Xie, “A question answering system based on mineral exploration ontology generation: A deep learning methodology”, Ore Geol. Rev., vol. 153, no. 105294, p. 105294, 2023.
15. [15] A. Al-Sadi, M. Al-Ayyoub, Y. Jararweh, and F. Costen, “Visual question answering in the medical domain based on deep learning approaches: A comprehensive study”, Pattern Recognit. Lett., vol. 150, pp. 57–75, 2021.
16. [16] N. Sharm, T. Jain, S. S. Narayan, and A. C. Kandakar, “Sentiment analysis of Amazon smartphone reviews using machine learning & deep learning”, in 2022 IEEE International Conference on Data Science and Information System (ICDSIS), 2022.
17. [17] D. Maity, S. Kanakaraddi, and S. Giraddi, “Text sentiment analysis based on multichannel convolutional neural networks and syntactic structure”, Procedia Comput. Sci., vol. 218, pp. 220–226, 2023.
18. [18] W. Li, L. Zhu, Y. Shi, K. Guo, and E. Cambria, “User reviews: Sentiment analysis using lexicon integrated two-channel CNN–LSTM family models”, Appl. Soft Comput., vol. 94, no. 106435, p. 106435, 2020.
19. [19] Y. Zhang, J. Wang, and X. Zhang, “Conciseness is better: Recurrent attention LSTM model for document-level sentiment analysis”, Neurocomputing, vol. 462, pp. 101–112, 2021.
20. [20] D. O. Oyewola, L. A. Oladimeji, S. O. Julius, L. B. Kachalla, and E. G. Dada, “Optimizing sentiment analysis of Nigerian 2023 presidential election using two-stage residual long short term memory”, Heliyon, vol. 9, no. 4, p. e14836, 2023.
21. [21] A. Patel, P. Oza, and S. Agrawal, “Sentiment analysis of customer feedback and reviews for airline services using language representation model”, Procedia Comput. Sci., vol. 218, pp. 2459–2467, 2023.
22. [22] M. P. Geetha and D. Karthika Renuka, “Improving the performance of aspect based sentiment analysis using fine-tuned Bert Base Uncased model”, International Journal of Intelligent Networks, vol. 2, pp. 64–69, 2021.
23. [23] A. Borg and M. Boldt, “Using VADER sentiment and SVM for predicting customer response sentiment”, Expert Syst. Appl., vol. 162, no. 113746, p. 113746, 2020.
24. [24] T. H. Jaya Hidayat, Y. Ruldeviyani, A. R. Aditama, G. R. Madya, A. W. Nugraha, and M. W. Adisaputra, “Sentiment analysis of twitter data related to Rinca Island development using Doc2Vec and SVM and logistic regression as classifier”, Procedia Comput. Sci., vol. 197, pp. 660–667, 2022.
25. [25] M. Bibi et al., “A novel unsupervised ensemble framework using concept-based linguistic methods and machine learning for twitter sentiment analysis”, Pattern Recognit. Lett., vol. 158, pp. 80–86, 2022.
26. [26] I. N. Khasanah, “Sentiment classification using fastText embedding and deep learning model”, Procedia Comput. Sci., vol. 189, pp. 343–350, 2021.
27. [27] P. F. Muhammad, R. Kusumaningrum, and A. Wibowo, “Sentiment analysis using Word2vec and long short-term memory (LSTM) for Indonesian hotel reviews”, Procedia Comput. Sci., vol. 179, pp. 728–735, 2021.
28. [28] K. Kaur and P. Kaur, “BERT-CNN: Improving BERT for requirements classification using CNN”, Procedia Comput. Sci., vol. 218, pp. 2604–2611, 2023.
29. [29] M. Siddharth and R. Aarthi, “Blended multi-class text to image synthesis GANs with RoBerTa and Mask R-CNN”, Procedia Comput. Sci., vol. 218, pp. 845–857, 2023.
30. [30] N. Badri, F. Kboubi, and A. H. Chaibi, “Combining FastText and glove word embedding for offensive and hate speech text detection”, Procedia Comput. Sci., vol. 207, pp. 769–778, 2022.
31. [31] K. Korovkinas, P. Danėnas, and G. Garšva, “SVM and k-means hybrid method for textual data sentiment analysis”, Balt. J. Mod. Comput., vol. 7, no. 1, 2019.
32. [32] A. S. M. AlQahtani, “Product Sentiment Analysis for Amazon Reviews”, Int. J. Comput. Sci. Inf. Technol., vol. 13, no. 3, pp. 15–30, 2021.
33. [33] S. A. Aljuhani and N. Saleh, “A comparison of sentiment analysis methods on Amazon reviews of mobile phones”, Int. J. Adv. Comput. Sci. Appl., vol. 10, no. 6, 2019.
34. [34] Sangeetha and Kumaran, ‘Sentiment analysis of amazon user reviews using a hybrid approach’, Measur. Sens., vol. 27, no. 100790, p. 100790, 2023.
35. [35] B. Bansal and S. Srivastava, “Sentiment classification of online consumer reviews using word vector representations”, Procedia Comput. Sci., vol. 132, pp. 1147–1153, 2018.
36. [36] L. Zhang, K. Hua, H. Wang, G. Qian, and L. Zhang, “Sentiment analysis on reviews of mobile users”, Procedia Comput. Sci., vol. 34, pp. 458–465, 2014.
37. [37] K. M. Karaoğlan and O. Fındık, “Extended rule-based opinion target extraction with a novel text pre-processing method and ensemble learning”, Appl. Soft Comput., vol. 118, no. 108524, p. 108524, 2022.
38. [38] A. K. Sharma, S. Chaurasia, and D. K. Srivastava, “Sentimental short sentences classification by using CNN deep learning model with fine tuned Word2Vec”, Procedia Comput. Sci., vol. 167, pp. 1139–1147, 2020.
39. [39] A. Pimpalkar and J. R. Raj R, “MBiLSTMGloVe: Embedding GloVe knowledge into the corpus using multi-layer BiLSTM deep learning model for social media sentiment analysis”, Expert Syst. Appl., vol. 203, no. 117581, p. 117581, 2022.
40. [40] M. Affi and C. Latiri, “BE-BLC: BERT-ELMO-based deep neural network architecture for English named entity recognition task”, Procedia Comput. Sci., vol. 192, pp. 168–181, 2021.
41. [41] A. Zhao and Y. Yu, “Knowledge-enabled BERT for aspect-based sentiment analysis”, Knowl. Based Syst., vol. 227, no. 107220, p. 107220, 2021.
42. [42] F. Gargiulo, S. Silvestri, M. Ciampi, and G. De Pietro, “Deep neural network for hierarchical extreme multi-label text classification”, Appl. Soft Comput., vol. 79, pp. 125–138, 2019.
43. [43] Z. A. Sejuti and M. S. Islam, “A hybrid CNN-KNN approach for identification of COVID-19 with 5-fold cross validation”, Sens. Int., vol. 4, no. 100229, p. 100229, 2023.