How Do Students Feel in Online Learning Platforms? How They Tell It: How Does Artificial Intelligence Make a Difference?

Year 2024, Volume: 14 Issue: 2 (Special Issue-Artificial Intelligence Tools and Education), 250 - 267, 26.07.2024

Bihter Daş Müzeyyen Bulut Özek Oğuzhan Özdemir

Abstract

This study aims to investigate the effectiveness of an artificial intelligence (AI) model in determining students' emotional states during online courses and compares these AI-generated results with traditional self-report methods used in educational sciences. Conducted with 66 students from three different departments of a public university in Eastern Turkey during the 2021-2022 academic year, the study involved capturing facial images of students every 10 minutes during online lectures to analyze their emotional states using a deep learning-based CNN model. In addition, students provided their emotional states through a mood analysis form, which included personal information and subjective feelings such as happiness, sadness, anger, and surprise. The AI model achieved a high accuracy rate of 90.12% in classifying seven different emotional states, demonstrating its potential for real-time emotion recognition in educational settings. However, the study also found a 39% overlap between AI-determined emotional states and self-reported emotions. This finding emphasizes the need for a multifaceted approach to emotion measurement, integrating both advanced AI techniques and traditional self-report tools to more comprehensively understand students' emotional experiences. The results highlight the challenges and opportunities in combining technology with educational assessments and suggest directions for future research in improving emotion detection methodologies and their application in online learning environments.

Keywords

Emotions, Sentiment Analysis, Artificial Intelligence, Online Learning, Self-Report

References

• Agrawal, A., & Mittal, N. (2020). Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. The Visual Computer, 36(2), 405-412. https://doi.org/10.1007/s00371-019-01630-9
• Aleven, V., McLaughlin, E.A., Glenn, R.A., & Koedinger, K.R. (2017). Instruction based on adaptive learning technologies. In: Handbook of Research on Learning and Instruction, (2nd ed.). pp. 522–560. New York: Routledge.
• Al‐Taweel, D., Al‐Haqan, A., Bajis, D., Al‐Bader, J., Al‐Taweel, A. M., Al‐Awadhi, A., & Al‐Awadhi, F. (2020). Multidisciplinary academic perspectives during the COVID‐19 pandemic. The International Journal of Health Planning and Management, 35(6), 1295-1301. https://doi.org/10.1002/hpm.3032
• Arzugül-Aksoy D., Bingöl, İ., & Bozkurt, A. (2022). Sorgulama Topluluğu Kuramı [Theory of Community of Inquiry]. Açık ve Uzaktan Öğrenme Kuramları [Open and Distance Learning Theories], Ankara: Nobel Akademik Yayıncılık.
• Bayrakçeken, S., Oktay, Ö., Samancı, O., & Canpolat, N. (2021). Motivasyon Kuramları Çerçevesinde Öğrencilerin Öğrenme Motivasyonlarının Arttırılması: Bir Derleme Çalışması [Increasing Students' Learning Motivation within the Framework of Motivation Theories: A Compilation Study]. Atatürk Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, 25(2), 677-698. Retrieved from https://dergipark.org.tr/en/pub/ataunisosbil/issue/62432/900664
• Berikan, B. (2020). Çevrimiçi öğrenme ortamlarında öğrenen-öğretim elemanı etkileşimi [Learner-instructor interaction in online learning environments]. Çevrimiçi Öğrenme Farklı Bakış Açıları, Ankara: Pegem Akademi.
• Bhardwaj, P., Gupta, P. K., Panwar, H., Siddiqui, M. K., Morales-Menendez, R., & Bhaik, A. (2021). Application of deep learning on student engagement in e-learning environments. Computers & Electrical Engineering, 93, 107277. https://doi.org/10.1016/j.compeleceng.2021.107277
• Boughida, A., Kouahla, M. N., & Lafifi, Y. (2022). A novel approach for facial expression recognition based on Gabor filters and genetic algorithm. Evolving Systems, 13(2), 331-345. https://doi.org/10.1007/s12530-021-09393-2
• Bouhlal, M., Aarika, K., Abdelouahid, R. A., Elfilali, S., & Benlahmar, E. (2020). Emotions recognition as innovative tool for improving students’ performance and learning approaches. Procedia Computer Science, 175, 597-602. https://doi.org/10.1016/j.procs.2020.07.086
• Bozkurt, A. (2020). Koronavirüs (Covid-19) pandemi süreci ve pandemi sonrası dünyada eğitime yönelik değerlendirmeler: Yeni normal ve yeni eğitim paradigması [Coronavirus (Covid-19) pandemic process and evaluations regarding education in the post-pandemic world: New normal and new education paradigm]. Açıköğretim Uygulamaları ve Araştırmaları Dergisi, 6(3), 112-142. Retrived from https://dergipark.org.tr/en/pub/auad/issue/56247/773769
• Bulut Özek, M. (2018). The effects of merging student emotion recognition with learning management systems on learners’ motivation and academic achievements. Computer applications in engineering education, 26(5), 1862-1872. https://doi.org/10.1002/cae.22000
• Castro, M.N., Vigob, D.E., Chu, E.M., Fahrer, R.D., Ach_a val, D., Costanzo, E.Y., Leiguarda, R.C., Nogu_e sa, M., Cardinali, D.P., & Guinjoan, S. M. (2009). Heart rate variability response to mental arithmetic stress is abnormal in first-degree relatives of individuals with schizophrenia. Schizophrenia Research, 109, 134–140. https://doi.org/10.1016/j.schres.2008.12.026
• Chandra, A., & Calderon, T. (2005). Challenges and constraints to the diffusion of biometrics in information systems. Communications of the ACM, 48 (12), 101–106. https://doi.org/10.1145/1101779.1101784
• Chen, J., Lv, Y., Xu, R. & Xu, C. (2019). Automatic social signal analysis: Facial expression recognition using difference convolution neural network. Journal of Parallel and Distributed Computing, 131, 97-102. https://doi.org/10.1016/j.jpdc.2019.04.017
• Chevalère, J., Lazarides, R., Yun, H. S., Henke, A., Lazarides, C., Pinkwart, N., & Hafner, V. V. (2023). Do instructional strategies considering activity emotions reduce students’ boredom in a computerized open-ended learning environment? Computers & Education, 196, 104741. https://doi.org/10.1016/j.compedu.2023.104741
• Chowdary, M. K., Nguyen, T. N., & Hemanth, D. J. (2021). Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Computing and Applications, 35, 23311–23328. https://doi.org/10.1007/s00521-021-06012-8
• Devi, S. A., & Ch, S. (2021). An efficient facial emotion recognition system using novel deep learning neural network-regression activation classifier. Multimedia Tools and Applications, 80(12), 17543-17568. https://doi.org/10.1007/s11042-021-10547-2
• D'mello, S., & Graesser, A. (2013). AutoTutor and affective AutoTutor: Learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(4), 1-39. https://doi.org/10.1145/2395123.2395128
• Do, L. N., Yang, H. J., Nguyen, H. D., Kim, S. H., Lee, G. S., & Na, I. S. (2021). Deep neural network-based fusion model for emotion recognition using visual data. The Journal of Supercomputing, 77, 10773–10790. https://doi.org/10.1007/s11227-021-03690-y
• Eliot, J. A., & Hirumi, A. (2019). Emotion theory in education research practice: An interdisciplinary critical literature review. Educational technology research and development, 67, 1065-1084. https://doi.org/10.1007/s11423-018-09642-3
• Fallahzadeh, M. R., Farokhi, F., Harimi, A., & Sabbaghi-Nadooshan, R. (2021). Facial Expression Recognition based on Image Gradient and Deep Convolutional Neural Network. Journal of AI and Data Mining, 9(2), 259-268. https://doi.org/10.22044/jadm.2021.9898.2121
• Gömleksiz, M. N., & Kan, A. Ü. (2012). Eğitimde duyuşsal boyut ve duyuşsal öğrenme [Affective dimension and affective learning in education]. Electronic Turkish Studies, 7(1), 1159-1177. Retrieved from https://www.ajindex.com/dosyalar/makale/acarindex-1423933804.pdf
• Graesser, A. C. (2020). Emotions are the experiential glue of learning environments in the 21st century. Learning and Instruction, 70, 101212. https://doi.org/10.1016/j.learninstruc.2019.05.009
• Gustiani, S. (2020). Students’ motivation in online learning during covıd-19 pandemic era: a case study. Holistics, 12(2), 23-40. Retrieved from https://jurnal.polsri.ac.id/index.php/holistic/article/view/3029
• Hasnine, M. N., Bui, H. T., Tran, T. T. T., Nguyen, H. T., Akçapınar, G., & Ueda, H. (2021). Students’ emotion extraction and visualization for engagement detection in online learning. Procedia Computer Science, 192, 3423-3431. https://doi.org/10.1016/j.procs.2021.09.115
• Hung, J. C., Lin, K. C., & Lai, N. X. (2019). Recognizing learning emotion based on convolutional neural networks and transfer learning. Applied Soft Computing, 84, 105724. https://doi.org/10.1016/j.asoc.2019.105724
• Imani, M., & Montazer, G. A. (2019). A survey of emotion recognition methods with emphasis on E-Learning environments. Journal of Network and Computer Applications, 147, 102423. https://doi.org/10.1016/j.jnca.2019.102423
• Jaiswal, S., & Nandi, G. C. (2020). Robust real-time emotion detection system using CNN architecture. Neural Computing and Applications, 32(15), 11253-11262. https://doi.org/10.1007/s00521-019-04564-4
• Kaddoura, S., & Gumaei, A. (2022). Towards effective and efficient online exam systems using deep learning-based cheating detection approach. Intelligent Systems with Applications, 16, 200153. https://doi.org/10.1016/j.iswa.2022.200153
• Khezri, M., Firoozabadi, M., & Sharafat, A. R. (2015). Reliable emotion recognition system based on dynamic adaptive fusion of forehead biopotentials and physiological signals. Computer methods and programs in biomedicine, 122(2), 149-164. https://doi.org/10.1016/j.cmpb.2015.07.006
• Kim, J. (2020). Learning and teaching online during Covid-19: Experiences of student teachers in an early childhood education practicum. International Journal of Early Childhood, 52(2), 145-158. https://doi.org/10.1007/s13158-020-00272-6
• Kumar, N. N., Summerell, E., Spehar, B., & Cranney, J. (2021). Experiences of Honours Research Students and Supervisors During the COVID-19 Pandemic: A Pilot Study Framed by Self-Determination Theory. In Frontiers in Education (p. 389). Frontiers.
• Lacave, C., Velázquez-Iturbide, J. Á., Paredes-Velasco, M., & Molina, A. I. (2020). Analyzing the influence of a visualization system on students' emotions: An empirical case study. Computers & Education, 149, 103817. https://doi.org/10.1016/j.compedu.2020.103817
• Li, B., & Lima, D. (2021). Facial expression recognition via ResNet-50. International Journal of Cognitive Computing in Engineering, 2, 57-64. https://doi.org/10.1016/j.ijcce.2021.02.002
• Liu, K., Zhang, M., & Pan, Z. (2016, September). Facial expression recognition with CNN ensemble. In 2016 international conference on cyberworlds (CW) (pp. 163-166). IEEE.
• Lyu, L., Zhang, Y., Chi, M. Y., Yang, F., Zhang, S. G., Liu, P., & Lu, W. G. (2022). Spontaneous facial expression database of learners’ academic emotions in online learning with hand occlusion. Computers & Electrical Engineering, 97, 107667. https://doi.org/10.1016/j.compeleceng.2021.107667
• Maqableh, W., Alzyoud, F. Y., & Zraqou, J. (2023). The use of facial expressions in measuring students’ interaction with distance learning environments during the COVID-19 crisis. Visual informatics, 7(1), 1-17. https://doi.org/10.1016/j.visinf.2022.10.001
• Martínez, F., Barraza, C., Gonzalez, N., & González, J. (2016). KAPEAN: Understanding Affective States of Children with ADHD. Educational Technology & Society, 19(2), 18–28. Retrieved from https://www.jstor.org/stable/pdf/jeductechsoci.19.2.18.pdf
• Mayer, R. E. (2020). Searching for the role of emotions in e-learning. Learning and Instruction, 70, 101213. https://doi.org/10.1016/j.learninstruc.2019.05.010
• Muhammad, G., & Hossain, M. S. (2021). Emotion recognition for cognitive edge computing using deep learning. IEEE Internet of Things Journal, 8(23), 16894-16901.
• Nicolaou, M. A., Gunes, H., & Pantic, M. (2012). Output-associative RVM regression for dimensional and continuous emotion prediction. Image and Vision Computing, 30(3), 186-196. https://doi.org/10.1016/j.imavis.2011.12.005
• Obergriesser, S., & Stoeger, H. (2020). Students’ emotions of enjoyment and boredom and their use of cognitive learning strategies–How do they affect one another?. Learning and Instruction, 66, 101285. https://doi.org/10.1016/j.learninstruc.2019.101285
• Öztüre, G., Fidan, A., Bakır, E., Uslu, N. A., & Usluel, Y. (2021). Eğitsel bağlamda teknoloji ve duygu çalışmaları üzerine bir sistematik haritalama çalışması: tanımlar, kuramlar ve gelecekteki yönelimler [A systematic mapping study on technology and emotion studies in educational contexts: definitions, theories and future directions.]. Eğitim Teknolojisi Kuram ve Uygulama, 11(1), 20-47. https://doi.org/10.17943/etku.745236
• Peng, X., & Xu, Q. (2020). Investigating learners' behaviors and discourse content in MOOC course reviews. Computers & Education, 143, 103673. https://doi.org/10.1016/j.compedu.2019.103673
• Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of educational Psychology, 95(4), 667. https://doi.org/10.1037/0022-0663.95.4.667
• Putwain, D. W., Becker, S., Symes, W., & Pekrun, R. (2018). Reciprocal relations between students’ academic enjoyment, boredom, and achievement over time. Learning and Instruction, 54, 73-81. https://doi.org/10.1016/j.learninstruc.2017.08.004
• Said, Y., & Barr, M. (2021). Human emotion recognition based on facial expressions via deep learning on high-resolution images. Multimedia Tools and Applications, 80(16), 25241-25253. https://doi.org/10.1007/s11042-021-10918-9
• Sakalle, A., Tomar, P., Bhardwaj, H., Acharya, D., & Bhardwaj, A. (2021). A LSTM based deep learning network for recognizing emotions using wireless brainwave driven system. Expert Systems with Applications, 173, 114516. https://doi.org/10.1016/j.eswa.2020.114516
• Sarrafzadeh, A., Alexander, S., Dadgostar, F., Fan, C., & Bigdeli, A. (2008). “How do you know that I don’t understand?” A look at the future of intelligent tutoring systems. Computers in Human Behavior, 24(4), 1342-1363. https://doi.org/10.1016/j.chb.2007.07.008
• Savci, P., & Das, B. (2023). Prediction of the customers’ interests using sentiment analysis in e-commerce data for comparison of Arabic, English, and Turkish languages. Journal of King Saud University - Computer and Information Sciences, 35(3), 227–237. https://doi.org/10.1016/j.jksuci.2023.02.017
• Scherer, K. R. (2005). What are emotions? And how can they be measured?. Social science information, 44(4), 695-729. https://doi.org/10.1177/0539018405058216
• Schneider, S., Krieglstein, F., Beege, M., & Rey, G. D. (2022). The impact of video lecturers’ nonverbal communication on learning–An experiment on gestures and facial expressions of pedagogical agents. Computers & Education, 176, 104350. https://doi.org/10.1016/j.compedu.2021.104350
• Sealy, M. K. (2021). Communication in the Time of COVID-19: An examination of imagined interactions and communication apprehension during the COVID-19 Pandemic. Imagination, Cognition and Personality, 41(2), 158-186. https://doi.org/10.1177/02762366211021076
• Sethi, K., & Jaiswal, V. (2022). PSU-CNN: prediction of student understanding in the classroom through student facial images using convolutional neural network. Materials Today: Proceedings, 62, 4957-4964. https://doi.org/10.1016/j.matpr.2022.03.691
• Sydänmaanlakka, A., Häsä, J., Holm, M. E., & Hannula, M. S. (2024). Mathematics-related achievement emotions – Interaction between learning environment and students’ mathematics performance. Learning and Individual Differences, 113, 102486. https://doi.org/10.1016/j.lindif.2024.102486
• Taub, M., Sawyer, R., Smith, A., Rowe, J., Azevedo, R., & Lester, J. (2020). The agency effect: The impact of student agency on learning, emotions, and problem-solving behaviors in a game-based learning environment. Computers & Education, 147, 103781. https://doi.org/10.1016/j.compedu.2019.103781
• Tonguç, G., & Ozkara, B. O. (2020). Automatic recognition of student emotions from facial expressions during a lecture. Computers & Education, 148, 103797. https://doi.org/10.1016/j.compedu.2019.103797
• Uçar, H. (2017). Açık ve uzaktan öğrenmede motivasyon tasarımı [Motivation design in open and distance learning]. V. Yüzer (Eds.) Açık ve Uzaktan Öğrenmede Bireysel Farklılıklar [Individual Differences in Open and Distance Learning] (pp.47-69). Anadolu Üniversitesi Açıköğretim Fakültesi Yayını.
• Wang, W., Xu, K., Niu, H., & Miao, X. (2020). [Retracted] Emotion Recognition of Students Based on Facial Expressions in Online Education Based on the Perspective of Computer Simulation. Complexity, 2020(1), 4065207. https://doi.org/10.1155/2020/4065207
• Yadegaridehkordi, E., Noor, N. F. B. M., Ayub, M. N. B., Affal, H. B., & Hussin, N. B. (2019). Affective computing in education: A systematic review and future research. Computers & Education, 142, 103649. https://doi.org/10.1016/j.compedu.2019.103649
• Yolcu, H. H. (2020). Koronavirüs (covid-19) pandemi sürecinde sınıf öğretmeni adaylarının uzaktan eğitim deneyimleri [Distance education experiences of classroom teacher candidates during the coronavirus (covid-19) pandemic]. Açıköğretim Uygulamaları ve Araştırmaları Dergisi, 6(4), 237-250. Retrieved from https://dergipark.org.tr/en/pub/auad/issue/57638/788890
• Yoshitomi, Y., Kim, S., Kawano, T., & Kilazoe, T. (2000). Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In 9th IEEE International Workshop on Robot and Human Interactive Communication, pp. 178–183.
• Yuan, Y., Li, Y., Nie, J., Chao, X., & Li, J. (2024). An individual adaptive ubiquitous learning paradigm: Focusing on the collection and utilization of academic emotions. Computers in Human Behavior, 156, 108228. https://doi.org/10.1016/j.chb.2024.108228
• Zembylas, M., Theodorou, M., & Pavlakis, A. (2008). The role of emotions in the experience of online learning: Challenges and opportunities. Educational Media International, 45(2), 107-117. https://doi.org/10.1080/09523980802107237