Machine Learning based Emotion classification in the COVID-19 Real World Worry Dataset

Abstract

COVID-19 pandemic has a dramatic impact on economies and communities all around the world. With social distancing in place and various measures of lockdowns, it becomes significant to understand emotional responses on a great scale. In this paper, a study is presented that determines human emotions during COVID-19 using various machine learning (ML) approaches. To this end, various techniques such as Decision Trees (DT), Support Vector Machines (SVM), k-nearest neighbor (k-NN), Neural Networks (NN) and Naïve Bayes (NB) methods are used in determination of the human emotions. The mentioned techniques are used on a dataset namely Real World Worry dataset (RWWD) that was collected during COVID-19. The dataset, which covers eight emotions on a 9-point scale, grading their anxiety levels about the COVID-19 situation, was collected by using 2500 participants. The performance evaluation of the ML techniques on emotion prediction is carried out by using the accuracy score. Five-fold cross validation technique is also adopted in experiments. The experiment works show that the ML approaches are promising in determining the emotion in COVID-19 RWWD. More specifically, the NN method produced the highest average accuracy scores for both emotion and gender classification where a 75.7% and 72.1% average scores were obtained.

Keywords

• COVID-19 worry dataset
• emotion classification
• machine learning

References

1. Agrawal, A., & An, A. (2012). Unsupervised emotion detection from text using semantic and syntactic relations. Proceedings - 2012 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2012, 346–353. https://doi.org/10.1109/WI-IAT.2012.170
2. Aha, D. W., Kibler, D., & Albert, M. K. (1991). Instance-based learning algorithms. Machine Learning, 6(1), 37–66. https://doi.org/10.1007/BF00153759
3. Akbulut, Y., Sengur, A., Guo, Y., & Smarandache, F. (2017). NS-k-NN: Neutrosophic set-based k-nearest neighbors classifier. Symmetry, 9(9). https://doi.org/10.3390/sym9090179
4. Alm, C. O., Roth, D., & Sproat, R. (2005). Emotions from text. October, 579–586. https://doi.org/10.3115/1220575.1220648
5. Altuntaş, Y., Kocamaz, A. F., Cömert, Z., Cengiz, R., & Esmeray, M. (2019). Identification of Haploid Maize Seeds using Gray Level Co-occurrence Matrix and Machine Learning Techniques. 2018 International Conference on Artificial Intelligence and Data Processing, IDAP 2018, 8–12. https://doi.org/10.1109/IDAP.2018.8620740
6. Bandhakavi, A., Wiratunga, N., Deepak, P., & Massie, S. (2014). Generating a word-emotion lexicon from #emotional tweets. Proceedings of the 3rd Joint Conference on Lexical and Computational Semantics, *SEM 2014, 12–21. https://doi.org/10.3115/v1/s14-1002
7. Bandhakavi, A., Wiratunga, N., Massie, S., & Padmanabhan, D. (2017). Lexicon Generation for Emotion Detection from Text. IEEE Intelligent Systems, 32(1), 102–108. https://doi.org/10.1109/MIS.2017.22
8. Bandhakavi, A., Wiratunga, N., Padmanabhan, D., & Massie, S. (2017). Lexicon based feature extraction for emotion text classification. Pattern Recognition Letters, 93, 133–142. https://doi.org/10.1016/j.patrec.2016.12.009

9. Boynukalin, Z. (2012). Emotion Analysis of Turkish Texts by Using Machine Learning Methods. Middle East Technical University.
10. Canales, L., & Martínez-Barco, P. (2015). Emotion Detection from text: A Survey. 37–43. https://doi.org/10.3115/v1/w14-6905
11. Chudacek, V., Spilka, J., Rubackova, B., Koucky, M., Georgoulas, G., Lhotska, L., & Stylios, C. (2008). Evaluation of feature subsets for classification of cardiotocographic recordings. Computers in Cardiology, 35(21), 845–848. https://doi.org/10.1109/CIC.2008.4749174
12. Cömert, Z., & Kocamaz, A. F. (2017). Comparison of machine learning techniques for fetal heart rate classification. Acta Physica Polonica A, 132(3), 451–454. https://doi.org/10.12693/APhysPolA.132.451
13. Fakhri, A., Nasir, A., Conf, I. O. P., Mater, S., Eng, S., Fakhri, A., Nasir, A., Nee, E. S., Choong, C. S., Shahrizan, A., Ghani, A., Majeed, A. P. P. A., Adam, A., & Furqan, M. (2020). Text-based emotion prediction system using machine learning approach. https://doi.org/10.1088/1757-899X/769/1/012022
14. Hasan, M., Agu, E., & Rundensteiner, E. (2014). Using Hashtags as Labels for Supervised Learning of Emotions in Twitter Messages. Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, 187–193.
15. Huang, M.-L., & Hsu, Y.-Y. (2012). Fetal distress prediction using discriminant analysis, decision tree, and artificial neural network. Journal of Biomedical Science and Engineering, 05(09), 526–533. https://doi.org/10.4236/jbise.2012.59065
16. Kleinberg, B., van der Vegt, I., & Mozes, M. (2020). Measuring Emotions in the COVID-19 Real World Worry Dataset. 1. http://arxiv.org/abs/2004.04225
17. Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2012). Foundations of Machine Learning. MIT Press.
18. Rey-Villamizar, N., Shrestha, P., Sadeque, F., Bethard, S., Pedersen, T., Mukherjee, A., & Solorio, T. (2016). Analysis of Anxious Word Usage on Online Health Forums. May 2017, 37–42. https://doi.org/10.18653/v1/w16-6105
19. Roberts, K., Roach, M. A., Johnson, J., Guthrie, J., & Harabagiu, S. M. (2012). EmpaTweet: Annotating and detecting emotions on twitter. Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012, 3806–3813.
20. Safavian, S. R., & Landgrebe, D. (1991). A survey of decision tree classifier methodology. IEEE Transactions on Systems, Man, and Cybernetics, 21(3), 660–674. https://doi.org/10.1109/21.97458
21. Sahin, H., & Subasi, A. (2015). Classification of the cardiotocogram data for anticipation of fetal risks using machine learning techniques. Applied Soft Computing Journal, 33, 231–238. https://doi.org/10.1016/j.asoc.2015.04.038
22. Seyeditabari, A., Tabari, N., & Zadrozny, W. (2018). Emotion Detection in Text: a Review. http://arxiv.org/abs/1806.00674
23. Suhasini, M., & Badugu, S. (2018). Two Step Approach for Emotion Detection on Twitter Data. International Journal of Computer Applications, 179(53), 12–19. https://doi.org/10.5120/ijca2018917350
24. Suttles, J., & Ide, N. (2013). Distant supervision for emotion classification with discrete binary values. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 7817 LNCS(PART 2), 121–136. https://doi.org/10.1007/978-3-642-37256-8_11
25. Taran, S., & Bajaj, V. (2019). Emotion recognition from single-channel EEG signals using a two-stage correlation and instantaneous frequency-based filtering method. Computer Methods and Programs in Biomedicine, 173, 157–165. https://doi.org/10.1016/j.cmpb.2019.03.015
26. van der Vegt, I., & Kleinberg, B. (2020). Women worry about family, men about the economy: Gender differences in emotional responses to COVID-19. 1–12. http://arxiv.org/abs/2004.08202
27. Wang, W., Chen, L., Thirunarayan, K., & Sheth, A. P. (2012). Harnessing twitter “big data” for automatic emotion identification. Proceedings - 2012 ASE/IEEE International Conference on Privacy, Security, Risk and Trust and 2012 ASE/IEEE International Conference on Social Computing, SocialCom/PASSAT 2012, 587–592. https://doi.org/10.1109/SocialCom-PASSAT.2012.119