Evaluating the Effectiveness of Different Machine Learning Approaches for Sentiment Classification

Year 2023, , 1496 - 1510, 01.09.2023

Seda Bayat , Gültekin Işık

https://doi.org/10.21597/jist.1292050

Abstract

This study presents a comparison of four different machine learning algorithms for sentiment analysis on a small subset of the AR-P (Amazon Reviews - Polarity) dataset. The algorithms evaluated are multilayer perceptron (MLP), Naive Bayes, Decision Tree, and Transformer architectures. The results show that the Transformer-based DistilBERT model performed the best with an accuracy rate of 96.10%, while MLP had a better performance than the other remaining methods. Confusion matrices and ROC curves are provided to illustrate the results, and a comparison with previous studies is presented. The study concludes that the results can serve as a basis for future work, such as using larger datasets or comparing the performance of algorithms on different tasks. Overall, this study provides insights into the use of traditional machine learning and modern deep learning methods for sentiment analysis and their potential applications in real-world scenarios.

Keywords

Deep Learning, Distilbert, Sentiment Analysis, Text Classification, Transformer

References

• Abdi, A., Shamsuddin, S. M., Hasan, S., & Piran, J. (2019). Deep learning-based sentiment classification of evaluative text based on Multi-feature fusion. Information Processing & Management, 56(4), 1245-1259.
• Ain, Q. T., Ali, M., Riaz, A., Noureen, A., Kamran, M., Hayat, B., & Rehman, A. (2017). Sentiment analysis using deep learning techniques: a review. International Journal of Advanced Computer Science and Applications, 8(6).
• Alexandridis, G., Varlamis, I., Korovesis, K., Caridakis, G., & Tsantilas, P. (2021). A survey on sentiment analysis and opinion mining in greek social media. Information, 12(8), 331.
• Al-Garadi, M. A., Yang, Y. C., Cai, H., Ruan, Y., O’Connor, K., Graciela, G. H., ... & Sarker, A. (2021). Text classification models for the automatic detection of nonmedical prescription medication use from social media. BMC Medical Informatics and Decision Making, 21(1), 1-13. DOI: 10.1186/s12911-021-01488-1
• Balahur, A., Turchi, M., & Steinberger, R. (2013). Multilingual sentiment analysis using machine translation–based techniques. ACM Transactions on Intelligent Systems and Technology (TIST), 4(1), 1-26. DOI: 10.1145/2444776.2444777
• Bayat, S., & Işık, G. (2022). Recognition of Aras Bird Species From Their Voices With Deep Learning Methods. Journal of the Institute of Science and Technology, 12(3), 1250-1263.
• Bollen, J., Mao, H., &. Zeng, X. (2011). Twitter mood predicts the stock market. Journal of Computational Science, 2(1), 1-8. https://doi.org/10.1016/j.jocs.2010.12.007
• Boyko, N., & Boksho, K. (2020, November). Application of the Naive Bayesian Classifier in Work on Sentimental Analysis of Medical Data. In Proceedings of the International Conference on Intelligent Data and Digital Medicine (IDDM) (pp. 230-239).
• Cambria, E., & White, B. (2014). Jumping NLP curves: A review of natural language processing research. IEEE Journal of Computational Intelligence, 9(2), 48-57. https://doi.org/10.1109/MCI.2014.2307227
• Chen, L. C., Lee, C. M., and Chen, M. Y. (2020) published a study in Soft Computing, in which they explored social media for sentiment analysis using deep learning techniques.
• Delobelle, P., Winters, T., & Berendt, B. (2020). Robbert: a dutch roberta-based language model. arXiv preprint arXiv:2001.06286.
• Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.
• Dogra, V., Verma, S., Kavita, C., Chatterjee, P., Shafi, J., Choi, J., & Ijaz, M. F. (2022). A Complete Process of Text Classification System Using State-of-the-Art NLP Models. Computational Intelligence and Neuroscience, 2022, 1883698. https://doi.org/10.1155/2022/1883698
• Gao, J., &. Wong, K.-F. (2014). A review of sentiment analysis research in Chinese language. Informatics, 1(3), 191-208. https://doi.org/10.3390/informatics1030191
• Ghulam, H., Zeng, F., Li, W., & Xiao, Y. (2019). Deep learning-based sentiment analysis for roman urdu text. Procedia computer science, 147, 131-135.
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
• Gupta, R., Sameer, S., Muppavarapu, H., Enduri, M. K., & Anamalamudi, S. (2021, September). Sentiment analysis on Zomato reviews. In 2021 13th International Conference on Computational Intelligence and Communication Networks (CICN) (pp. 34-38). IEEE.
• Gündüz, M. Ş., & Işık, G. (2023). A new YOLO-based method for social distancing from real-time videos. Neural Computing and Applications, 1-11.
• Han, J., & Kamber, M. (2011). Data mining: concepts and techniques. Elsevier.
• Hugging Face. (2023b). AutoModelForSequenceClassification. https://huggingface.co/transformers/model_doc/auto.html#automodelforsequenceclassification
• Hutto, C. J., & Gilbert, E. (2014). VADER: A parsimonious rule-based model for sentiment analysis of social media text. Eighth International AAAI Conference on Weblogs and Social Media. https://www.aaai.org/ocs/index.php/ICWSM/ICWSM14/paper/view/8109/8122
• Joshi, A., Bhattacharyya, P., & Carman, M. J. (2020). A thorough examination of the DistilBERT model for sentence classification. arXiv preprint arXiv:2010.16061.
• Kim, S. (2020). Sentiment analysis: A comprehensive guide to detecting emotions, opinions, and sentiments.
• Kumar, V. (2022). A Review of Decision Tree Algorithms for Classification in Machine Learning. International Journal of Computer Applications, 182(40), 10-16.
• Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2020). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv preprint arXiv:1909.11942.
• Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.
• M. Al-Qurishi, T. Khalid and R. Souissi, "Deep Learning for Sign Language Recognition: Current Techniques, Benchmarks, and Open Issues," in IEEE Access, vol. 9, pp. 126917-126951, 2021, doi: 10.1109/ACCESS.2021.3110912.
• Mohammad, S. M., & Bravo-Marquez, F. (2017). WASSA-2017 shared task on emotion intensity. Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, 1-10. https://doi.org/10.18653/v1/W17-5201
• Nandwani, P., & Verma, R. (2021). A review on sentiment analysis and emotion detection from text. Social Network Analysis and Mining, 11(1), 81.
• Othan, D., Kilimci, Z. H., & Uysal, M. (2019, December). Financial sentiment analysis for predicting direction of stocks using bidirectional encoder representations from transformers (BERT) and deep learning models. In Proc. Int. Conf. Innov. Intell. Technol (pp. 30-35).
• Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2), 1-135. https://doi.org/10.1561/1500000011
• Poria, S., Cambria, E., &. Bajpai, R. (2017). A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98-125. https://doi.org/10.1016/j.inffus.2017.02.003
• Raschka, S., & Mirjalili, V. (2021). Naive Bayes and Text Classification. In Python Machine Learning, Third Edition (pp. 373-394). Packt Publishing.
• Ray, P., & Chakrabarti, A. (2022). A mixed approach of deep learning method and rule-based method to improve aspect level sentiment analysis. Applied Computing and Informatics, 18(1/2), 163-178.
• Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
• Sudhir, P., & Suresh, V. D. (2021). Comparative study of various approaches, applications and classifiers for sentiment analysis. Global Transitions Proceedings, 2(2), 205-211.
• Sun, C., Li, L., Wang, W., &. Jiang, B. (2021). Multi-task learning for sentiment analysis using transformer- Khalid based models. Neural Networks, 137, 181-190. https://doi.org/10.1016/j.neunet.2020.11.010
• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998-6008.
• Wankhade, M., Rao, A. C. S., & Kulkarni, C. (2022). A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7), 5731-5780.