Automated Detection of Fundamental Emotions from Literary Texts of Middle Grade Students

Abstract

The middle school period covers the beginning of the adolescent years of individuals. In this period, children display very complex and chaotic negative emotions such as anxiety, shyness, loneliness, guilt, depression, and anger intensely. Detecting these emotions accurately can help facilitate coping with the crises of this period and regulate emotions. It also can prevent potential suffering in later ages, reduce societal costs and prevent secondary diagnosis from developing. Emotions are by their nature at the center of literature and are better understood through the experience of writing or reading literary texts. Studies also show that the best type of written text to analyze for detection of emotions is informal texts. We believe the literary value and spontaneity of texts written by children without any given topic allow them to better channel their fundamental emotions into these texts without any reservations. In this study, we designed and developed a website for middle grade students to upload their literary works such as poems, fairytales, stories, etc. and collected and analyzed their texts to identify five fundamental emotions (anger, fear, disgust, sadness, and joy) using sentiment analysis methods and machine learning models. We use text representation techniques such as Bag-Of-Words, TF-IDF, and Word2Vector and neural network models such as Multi-layer Perceptrons (MLPs), and Convolutional Neural Networks (CNNs). We obtained the best results with TF-IDF + MLP model in terms of average f1-score, precision and recall metrics (0.73, 0.71, 0.83) and the best detected emotion is Joy while the worst detected is Disgust. We believe the accuracy of our models can increase with the inclusion of more data and promoting and making the website more accessible can allow us to enrich our dataset. Moreover, after the model is tested again with a larger sample, this system can be integrated with guidance and psychological counseling services, paving the way for psychosocial support needs analysis for students. A systematic monitoring mechanism could allow for the tracking of students' emotional changes over time and help develop early intervention strategies.

Keywords

• sentiment analysis
• emotion detection
• neural networks
• natural language processing
• adolescent psychology

References

1. Acheampong, F. A., Nunoo-Mensah, H., & Chen, W. (2021). Transformer models for text-based emotion detection: A review. IEEE Access, 9, 182760–182777. https://doi.org/10.1109/ACCESS.2021.3112728
2. Akın, A. A., & Akın, M. D. (2007). Zemberek: An open source NLP framework for Turkic languages. Structure, 10, 1–5. https://doi.org/10.1.1.556.69
3. Bauer, D. (1976). An exploratory study of developmental changes in children's fears. Journal of child psychology and psychiatry, and allied disciplines, 17 1, 69-74. https://doi.org/10.1111/j.1469-7610.1976.tb00375.x.
4. Bhargava, C., Poornima, S., Mahur, S., & Pushpalatha, M. (2021). Depression detection using sentiment analysis of tweets. Turkish Journal of Computer and Mathematics Education, 12(11), 5411–5418. https://turcomat.org/index.php/turkbilmat/article/view/6770
5. Boyacıoğlu, N. E., & Küçük, L. (2011). How do irrational beliefs affect test anxiety during adolescence? Journal of Psychiatric Nursing, 2(1), 40–45. https://phdergi.org/jvi.aspx?un=PHD-03164&volume=2&issue=1
6. Callaghan, B., Meyer, H., Opendak, M., Van Tieghem, M., Harmon, C., Li, A., Lee, F., Sullivan, R., & Tottenham, N. (2019). Using a Developmental Ecology Framework to Align Fear Neurobiology Across Species. Annual review of clinical psychology, 15, 345-369. https://doi.org/10.1146/annurev-clinpsy-050718-095727.
7. Camacho-Morles, J., Slemp, G., Pekrun, R., Loderer, K., Hou, H., & Oades, L. (2021). Activity Achievement Emotions and Academic Performance: A Meta-analysis. Educational Psychology Review, 33, 1051 - 1095. https://doi.org/10.1007/s10648-020-09585-3.
8. Charte, F., Rivera, A. J., del Jesús, M. J., & Herrera, F. (2015). MLSMOTE: Approaching imbalanced multilabel learning through synthetic instance generation. Knowledge-Based Systems, 89, 385–397. https://doi.org/10.1016/j.knosys.2015.07.019

9. Cormier, S., & Hackney, H. L. (2013). Counseling strategies and interventions: Pearson New International Edition. Pearson Higher Education.
10. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), 4171–4186. https://doi.org/10.18653/v1/N19-1423
11. Dong, Y., Hsiao, Y., Dawson, N., Banerji, N., & Nation, K. (2024). The emotional content of children’s writing: A data-driven approach. Cognitive Science, 48(3), e13423. https://doi.org/10.1111/cogs.13423
12. Egbochuku, E. O., Obodo, B., & Obadan, N. O. (2008). Efficacy of rational-emotive behaviour therapy on the reduction of test anxiety among adolescents in secondary schools. European Journal of Social Sciences, 6(4), 152–164. https://www.researchgate.net/publication/254064865_Efficacy_of_Rational-Emotive_Behaviour_Therapy_on_the_Reduction_of_Test_Anxiety_Among_Adolescents_in_Secondary_Schools
13. Ekman, P. (1992). Are there basic emotions? Psychological Review, 99(3), 550–553. https://doi.org/10.1037/0033-295X.99.3.550
14. Ekman, P. (2004). What we become emotional about. In A. S. R. Manstead, N. H. Frijda, & A. H. Fischer (Eds.), Feelings and emotions: The Amsterdam symposium (pp. 119–135). Cambridge University Press. https://doi.org/10.1017/CBO9780511806582.008
15. Ekman, P. (2016). What scientists who study emotion agree about. Perspectives on Psychological Science, 11(1), 31–34. https://doi.org/10.1177/1745691615596992
16. Goleman, D. (1998). Duygusal zeka. İstanbul: Varlık Yayınları, 34. https://www.varlik.com.tr/kitapDetay.aspx?kategoriID=16&kitapID=49
17. Google. (2024). Google news dataset.
19. Göçen, G. (2019). The effect of creative writing activities on elementary school students’ creative writing achievement, writing attitude and motivation. Journal of Language and Linguistic Studies, 15(3), 1032-1044.
20. Greenberg, L. S., & Pascual-Leone, A. (2006). Emotion in psychotherapy: A practice-friendly research review. Journal of Clinical Psychology, 62 (5), 611–630. https://doi.org/10.1002/jclp.20252
21. Haggren, H., & Amethier, P. (2018). Data-driven predictions of outcome for an internet-delivered treatment against anxiety disorders: A comparison of clinician and algorithm performance (bachelor’s thesis, KTH Royal Institute of Technology). Diva Portal. https://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-230705
22. Havigerová, J. M., Haviger, J., Kučera, D., & Hoffmannová, P. (2019). Text-based detection of the risk of depression. Frontiers in Psychology, 10, 513. https://doi.org/10.3389/fpsyg.2019.00513
23. Hipson, W. (2019). Using sentiment analysis to detect affect in children’s and adolescents’ poetry. International Journal of Behavioral Development, 43, 375 - 382. https://doi.org/10.1177/0165025419830248.
24. Hofmann, J., Troiano, E., Sassenberg, K., & Klinger, R. (2020). Appraisal Theories for Emotion Classification in Text. Proceedings of the 28th International Conference on Computational Linguistics, 125–138.
25. Jacobs, A. M., Herrmann, B., Lauer, G., Lüdtke, J., & Schroeder, S. (2020). Sentiment analysis of children and youth literature: Is there a pollyanna effect? Frontiers in Psychology, 11, 574746. https://doi.org/10.3389/fpsyg.2020.574746
26. Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017). Bag of tricks for efficient text classification. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017), 427–431. https://doi.org/10.18653/v1/E17-2068
27. Kauschke, C., Bahn, D., Vesker, M., & Schwarzer, G. (2019). The Role of Emotional Valence for the Processing of Facial and Verbal Stimuli—Positivity or Negativity Bias? Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.01654.
28. Kırmızı, F., & Adıgüzel, D. Ç. (2023). İlkokul öğrencilerinin yaratıcı yazma ürünlerinin bazı değişkenlere göre değerlendirilmesi. Pamukkale Üniversitesi Eğitim Fakültesi Dergisi, 58, 370-397.
29. Kidd, D. C., & Castano, E. (2013). Reading literary fiction improves theory of mind. Science, 342(6156), 377–380. https://doi.org/10.1126/science.1239918
30. Kim, S., Valitutti, A., & Calvo, R. (2010). Evaluation of Unsupervised Emotion Models to Textual Affect Recognition. , 62-70.
31. Krommyda, M., Rigos, A., Bouklas, K., & Amditis, A. (2021). An Experimental Analysis of Data Annotation Methodologies for Emotion Detection in Short Text Posted on Social Media. Informatics, 8, 19. https://doi.org/10.3390/informatics8010019.
32. Kumschick, I., Beck, L., Eid, M., Witte, G., Klann-Delius, G., Heuser, I., Steinlein, R., & Menninghaus, W. (2014). Reading and Feeling: The effects of a literature-based intervention designed to increase emotional competence in second and third graders. Frontiers in Psychology, 5, 1448. https://doi.org/10.3389/fpsyg.2014.01448.
33. Makinist, S., Hallaç, İ. R., Karakuş, B. A., & Aydın, G. (2017). Preparation of improved Turkish dataset for sentiment analysis in social media. ITM Web of Conferences, 13, 01030. https://doi.org/10.1051/itmconf/20171301030
34. Mohsin, M. A., & Beltiukov, A. (2019, May). Summarizing emotions from text using Plutchik’s wheel of emotions. In Proceedings of the 7th Scientific Conference on Information Technologies for Intelligent Decision-Making Support (ITIDS 2019) (pp. 291–294). Atlantis Press. https://doi.org/10.2991/itids-19.2019.52
35. Nook, E., Stavish, C., Sasse, S., Lambert, H., Mair, P., McLaughlin, K., & Somerville, L. (2020). Charting the development of emotion comprehension and abstraction from childhood to adulthood using observer-rated and linguistic measures. Emotion. 20(5), 773-792. https://doi.org/10.1037/emo0000609.
36. Oatley, K. (2002). Emotions and the story worlds of fiction. In M. C. Green, J. J. Strange, & T. C. Brock (Eds.), Narrative impact: Social and cognitive foundations (pp. 39–69). Lawrence Erlbaum Associates. https://www.routledge.com/Narrative-Impact-Social-and-Cognitive-Foundations/Green-Strange-Brock/p/book/9780805832990
37. Orhan, Z., Mercan, M., & Sertbaş, A. (2014). Turkish text analysis system for automatic detection of psychiatric disorders. Türkiye Bilişim Vakfı Bilgisayar Bilimleri ve Mühendisliği Dergisi, 7(1), 24–30. https://dergipark.org.tr/en/pub/tbbmd/issue/22247/238819
38. Özgüle, E. T. U., & Sümer, N. (2017). Ergenlikte duygu düzenleme ve psikolojik uyum: Duygu düzenleme ölçeğinin Türkçe uyarlaması. Turk Psikoloji Yazıları, 20 (40), 1–18. https://dergipark.org.tr/tr/download/article-file/2133036
39. Panksepp, J. (2004). Affective neuroscience: The foundations of human and animal emotions. Oxford University Press. https://global.oup.com/academic/product/affective-neuroscience-9780195178050
40. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311–318). Association for Computational Linguistics. https://doi.org/10.3115/1073083.1073135
41. Plutchik, R. (2001). Integration, differentiation, and derivatives of emotion. Evolution and Cognition, 7(2)), 114–125. https://doi.org/10.1007/978-3-319-28099-8_547-1
42. Provoost, S., Ruwaard, J., Van Breda, W., Riper, H., & Bosse, T. (2019). Validating automated sentiment analysis of online cognitive behavioral therapy patient texts: An exploratory study. Frontiers in psychology, 10, 1065. https://doi.org/10.3389/fpsyg.2019.01065
43. Puspitoningrum, E., Khan, R., & Swanjaya, D. (2024). Creative Writing as an Effort to Improve Children's Emotional Intelligence. Qubahan Academic Journal. https://doi.org/10.48161/qaj.v4n3a695.
44. Rutter, M. J., Bishop, D., Pine, D. S., Scott, S., Stevenson, J. S., Taylor, E. A., & Thapar, A. (2011). Rutter’s child and adolescent psychiatry (5th edition). John Wiley & Sons.
45. Shirazi, Y., Punamäki, R. L., Peltonen, K., Malekzadeh, M., & Esmaeili, O. (2021). Narrative-based intervention and emotional intelligence in female children. East European Journal of Psycholinguistics, 8(1), 40–55. https://doi.org/10.29038/eejpl.2021.8.1.shi
46. Stephen, J. J., & Prabu, P. (2019). Detecting the magnitude of depression in twitter users using sentiment analysis. International Journal of Electrical and Computer Engineering (IJECE), 9(4), 3247. https://doi.org/10.11591/ijece.v9i4.pp3247-3255
47. Tanju, E. H. (2010). Çocuklarda kitap okuma alışkanlığı’na genel bir bakış. Sosyal Politika Çalışmaları Dergisi, 21(21), 30–39. https://doi.org/10.21560/spcd.43960
48. Thompson, R. A. (2007). Socialization of emotion regulation in the family. In J. J. Gross (Ed.), Handbook of emotion regulation (pp. 249–268). The Guilford Press.
49. Tokcaer, S. (2021). Türkçe metinlerde duygu analizi. Yaşar Üniversitesi E-Dergisi, 16 (63), 1514–1534. https://doi.org/10.19168/jyasar.928843
50. Türkben, T. (2021). Ortaokul öğrencilerininn yazma becerileri, yazmaya yönelik tutumlari ve yazma motivasyonlari arasindaki ilişki. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 21(3), 900–922. https://doi.org/10.17240/aibuefd.2021.21.64908-788146
51. Uymaz, H., & Metin, S. (2023). Enriching Transformer-Based Embeddings for Emotion Identification in an Agglutinative Language: Turkish. IT Professional, 25, 67-73. https://doi.org/10.1109/mitp.2023.3278029.
52. Villani, D., Carissoli, C., Triberti, S., Marchetti, A., Gilli, G., & Riva, G. (2018). Videogames for Emotion Regulation: A Systematic Review. Games for Health Journal, 7 2, 85-99. https://doi.org/10.1089/g4h.2017.0108.
53. Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2018). Predicting suicide attempts in adolescents with longitudinal clinical data and machine learning. Journal of child psychology and psychiatry, 59(12), 1261–1270. https://doi.org/10.1111/jcpp.12916
54. Wang, S., & Manning, C. D. (2012). Baselines and bigrams: Simple, good sentiment and topic classification. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012), 90–94.
55. Wang, X., Zhang, C., Ji, Y., Sun, L., Wu, L., & Bao, Z. (2013). A depression detection model based on sentiment analysis in micro-blog social network. In L. Liu, J. Qu, & C. S. Jensen (Eds.), Trends and applications in knowledge discovery and data mining: PAKDD 2013 international workshops: DMApps, DANTH, QIMIE, BDM, CDA, CloudSD, Gold Coast, QLD, Australia, April 14-17, 2013, revised selected papers (Vol. 7867, pp. 201–213). Springer. https://doi.org/10.1007/978-3-642-38614-7_19
56. Weismayer, C., Gunter, U., & Önder, I. (2021). Temporal variability of emotions in social media posts. Technological Forecasting and Social Change, 167, 120699. https://doi.org/10.1016/j.techfore.2021.120699
57. Yiend, J., Mackintosh, B., & Mathews, A. (2005). Enduring consequences of experimentally induced biases in interpretation. Behaviour Research and Therapy, 43 (6), 779–797. https://doi.org/10.1016/j.brat.2004.05.003
58. Zhou, D., Zhang, X., Zhou, Y., Zhao, Q., & Geng, X. (2016, November). Emotion distribution learning from texts. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (pp. 638-647). Association for Computational Linguistics. https://doi.org/10.18653/v1/D16-1061