How Do Students Feel in Online Learning Platforms? How They Tell It: How Does Artificial Intelligence Make a Difference?

Abstract

This study aims to investigate the effectiveness of an artificial intelligence (AI) model in determining students' emotional states during online courses and compares these AI-generated results with traditional self-report methods used in educational sciences. Conducted with 66 students from three different departments of a public university in Eastern Turkey during the 2021-2022 academic year, the study involved capturing facial images of students every 10 minutes during online lectures to analyze their emotional states using a deep learning-based CNN model. In addition, students provided their emotional states through a mood analysis form, which included personal information and subjective feelings such as happiness, sadness, anger, and surprise. The AI model achieved a high accuracy rate of 90.12% in classifying seven different emotional states, demonstrating its potential for real-time emotion recognition in educational settings. However, the study also found a 39% overlap between AI-determined emotional states and self-reported emotions. This finding emphasizes the need for a multifaceted approach to emotion measurement, integrating both advanced AI techniques and traditional self-report tools to more comprehensively understand students' emotional experiences. The results highlight the challenges and opportunities in combining technology with educational assessments and suggest directions for future research in improving emotion detection methodologies and their application in online learning environments.

Keywords

• Emotions
• Sentiment Analysis
• Artificial Intelligence
• Online Learning
• Self-Report

References

1. Agrawal, A., & Mittal, N. (2020). Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. The Visual Computer, 36(2), 405-412. https://doi.org/10.1007/s00371-019-01630-9
2. Aleven, V., McLaughlin, E.A., Glenn, R.A., & Koedinger, K.R. (2017). Instruction based on adaptive learning technologies. In: Handbook of Research on Learning and Instruction, (2nd ed.). pp. 522–560. New York: Routledge.
3. Al‐Taweel, D., Al‐Haqan, A., Bajis, D., Al‐Bader, J., Al‐Taweel, A. M., Al‐Awadhi, A., & Al‐Awadhi, F. (2020). Multidisciplinary academic perspectives during the COVID‐19 pandemic. The International Journal of Health Planning and Management, 35(6), 1295-1301. https://doi.org/10.1002/hpm.3032
4. Arzugül-Aksoy D., Bingöl, İ., & Bozkurt, A. (2022). Sorgulama Topluluğu Kuramı [Theory of Community of Inquiry]. Açık ve Uzaktan Öğrenme Kuramları [Open and Distance Learning Theories], Ankara: Nobel Akademik Yayıncılık.
5. Bayrakçeken, S., Oktay, Ö., Samancı, O., & Canpolat, N. (2021). Motivasyon Kuramları Çerçevesinde Öğrencilerin Öğrenme Motivasyonlarının Arttırılması: Bir Derleme Çalışması [Increasing Students' Learning Motivation within the Framework of Motivation Theories: A Compilation Study]. Atatürk Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, 25(2), 677-698. Retrieved from https://dergipark.org.tr/en/pub/ataunisosbil/issue/62432/900664
6. Berikan, B. (2020). Çevrimiçi öğrenme ortamlarında öğrenen-öğretim elemanı etkileşimi [Learner-instructor interaction in online learning environments]. Çevrimiçi Öğrenme Farklı Bakış Açıları, Ankara: Pegem Akademi.
7. Bhardwaj, P., Gupta, P. K., Panwar, H., Siddiqui, M. K., Morales-Menendez, R., & Bhaik, A. (2021). Application of deep learning on student engagement in e-learning environments. Computers & Electrical Engineering, 93, 107277. https://doi.org/10.1016/j.compeleceng.2021.107277
8. Boughida, A., Kouahla, M. N., & Lafifi, Y. (2022). A novel approach for facial expression recognition based on Gabor filters and genetic algorithm. Evolving Systems, 13(2), 331-345. https://doi.org/10.1007/s12530-021-09393-2

9. Bouhlal, M., Aarika, K., Abdelouahid, R. A., Elfilali, S., & Benlahmar, E. (2020). Emotions recognition as innovative tool for improving students’ performance and learning approaches. Procedia Computer Science, 175, 597-602. https://doi.org/10.1016/j.procs.2020.07.086
10. Bozkurt, A. (2020). Koronavirüs (Covid-19) pandemi süreci ve pandemi sonrası dünyada eğitime yönelik değerlendirmeler: Yeni normal ve yeni eğitim paradigması [Coronavirus (Covid-19) pandemic process and evaluations regarding education in the post-pandemic world: New normal and new education paradigm]. Açıköğretim Uygulamaları ve Araştırmaları Dergisi, 6(3), 112-142. Retrived from https://dergipark.org.tr/en/pub/auad/issue/56247/773769
11. Bulut Özek, M. (2018). The effects of merging student emotion recognition with learning management systems on learners’ motivation and academic achievements. Computer applications in engineering education, 26(5), 1862-1872. https://doi.org/10.1002/cae.22000
12. Castro, M.N., Vigob, D.E., Chu, E.M., Fahrer, R.D., Ach_a val, D., Costanzo, E.Y., Leiguarda, R.C., Nogu_e sa, M., Cardinali, D.P., & Guinjoan, S. M. (2009). Heart rate variability response to mental arithmetic stress is abnormal in first-degree relatives of individuals with schizophrenia. Schizophrenia Research, 109, 134–140. https://doi.org/10.1016/j.schres.2008.12.026
13. Chandra, A., & Calderon, T. (2005). Challenges and constraints to the diffusion of biometrics in information systems. Communications of the ACM, 48 (12), 101–106. https://doi.org/10.1145/1101779.1101784
14. Chen, J., Lv, Y., Xu, R. & Xu, C. (2019). Automatic social signal analysis: Facial expression recognition using difference convolution neural network. Journal of Parallel and Distributed Computing, 131, 97-102. https://doi.org/10.1016/j.jpdc.2019.04.017
15. Chevalère, J., Lazarides, R., Yun, H. S., Henke, A., Lazarides, C., Pinkwart, N., & Hafner, V. V. (2023). Do instructional strategies considering activity emotions reduce students’ boredom in a computerized open-ended learning environment? Computers & Education, 196, 104741. https://doi.org/10.1016/j.compedu.2023.104741
16. Chowdary, M. K., Nguyen, T. N., & Hemanth, D. J. (2021). Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Computing and Applications, 35, 23311–23328. https://doi.org/10.1007/s00521-021-06012-8
17. Devi, S. A., & Ch, S. (2021). An efficient facial emotion recognition system using novel deep learning neural network-regression activation classifier. Multimedia Tools and Applications, 80(12), 17543-17568. https://doi.org/10.1007/s11042-021-10547-2
18. D'mello, S., & Graesser, A. (2013). AutoTutor and affective AutoTutor: Learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(4), 1-39. https://doi.org/10.1145/2395123.2395128
19. Do, L. N., Yang, H. J., Nguyen, H. D., Kim, S. H., Lee, G. S., & Na, I. S. (2021). Deep neural network-based fusion model for emotion recognition using visual data. The Journal of Supercomputing, 77, 10773–10790. https://doi.org/10.1007/s11227-021-03690-y
20. Eliot, J. A., & Hirumi, A. (2019). Emotion theory in education research practice: An interdisciplinary critical literature review. Educational technology research and development, 67, 1065-1084. https://doi.org/10.1007/s11423-018-09642-3
21. Fallahzadeh, M. R., Farokhi, F., Harimi, A., & Sabbaghi-Nadooshan, R. (2021). Facial Expression Recognition based on Image Gradient and Deep Convolutional Neural Network. Journal of AI and Data Mining, 9(2), 259-268. https://doi.org/10.22044/jadm.2021.9898.2121
22. Gömleksiz, M. N., & Kan, A. Ü. (2012). Eğitimde duyuşsal boyut ve duyuşsal öğrenme [Affective dimension and affective learning in education]. Electronic Turkish Studies, 7(1), 1159-1177. Retrieved from https://www.ajindex.com/dosyalar/makale/acarindex-1423933804.pdf
23. Graesser, A. C. (2020). Emotions are the experiential glue of learning environments in the 21st century. Learning and Instruction, 70, 101212. https://doi.org/10.1016/j.learninstruc.2019.05.009
24. Gustiani, S. (2020). Students’ motivation in online learning during covıd-19 pandemic era: a case study. Holistics, 12(2), 23-40. Retrieved from https://jurnal.polsri.ac.id/index.php/holistic/article/view/3029
25. Hasnine, M. N., Bui, H. T., Tran, T. T. T., Nguyen, H. T., Akçapınar, G., & Ueda, H. (2021). Students’ emotion extraction and visualization for engagement detection in online learning. Procedia Computer Science, 192, 3423-3431. https://doi.org/10.1016/j.procs.2021.09.115
26. Hung, J. C., Lin, K. C., & Lai, N. X. (2019). Recognizing learning emotion based on convolutional neural networks and transfer learning. Applied Soft Computing, 84, 105724. https://doi.org/10.1016/j.asoc.2019.105724
27. Imani, M., & Montazer, G. A. (2019). A survey of emotion recognition methods with emphasis on E-Learning environments. Journal of Network and Computer Applications, 147, 102423. https://doi.org/10.1016/j.jnca.2019.102423
28. Jaiswal, S., & Nandi, G. C. (2020). Robust real-time emotion detection system using CNN architecture. Neural Computing and Applications, 32(15), 11253-11262. https://doi.org/10.1007/s00521-019-04564-4
29. Kaddoura, S., & Gumaei, A. (2022). Towards effective and efficient online exam systems using deep learning-based cheating detection approach. Intelligent Systems with Applications, 16, 200153. https://doi.org/10.1016/j.iswa.2022.200153
30. Khezri, M., Firoozabadi, M., & Sharafat, A. R. (2015). Reliable emotion recognition system based on dynamic adaptive fusion of forehead biopotentials and physiological signals. Computer methods and programs in biomedicine, 122(2), 149-164. https://doi.org/10.1016/j.cmpb.2015.07.006
31. Kim, J. (2020). Learning and teaching online during Covid-19: Experiences of student teachers in an early childhood education practicum. International Journal of Early Childhood, 52(2), 145-158. https://doi.org/10.1007/s13158-020-00272-6
32. Kumar, N. N., Summerell, E., Spehar, B., & Cranney, J. (2021). Experiences of Honours Research Students and Supervisors During the COVID-19 Pandemic: A Pilot Study Framed by Self-Determination Theory. In Frontiers in Education (p. 389). Frontiers.
33. Lacave, C., Velázquez-Iturbide, J. Á., Paredes-Velasco, M., & Molina, A. I. (2020). Analyzing the influence of a visualization system on students' emotions: An empirical case study. Computers & Education, 149, 103817. https://doi.org/10.1016/j.compedu.2020.103817
34. Li, B., & Lima, D. (2021). Facial expression recognition via ResNet-50. International Journal of Cognitive Computing in Engineering, 2, 57-64. https://doi.org/10.1016/j.ijcce.2021.02.002
35. Liu, K., Zhang, M., & Pan, Z. (2016, September). Facial expression recognition with CNN ensemble. In 2016 international conference on cyberworlds (CW) (pp. 163-166). IEEE.
36. Lyu, L., Zhang, Y., Chi, M. Y., Yang, F., Zhang, S. G., Liu, P., & Lu, W. G. (2022). Spontaneous facial expression database of learners’ academic emotions in online learning with hand occlusion. Computers & Electrical Engineering, 97, 107667. https://doi.org/10.1016/j.compeleceng.2021.107667
37. Maqableh, W., Alzyoud, F. Y., & Zraqou, J. (2023). The use of facial expressions in measuring students’ interaction with distance learning environments during the COVID-19 crisis. Visual informatics, 7(1), 1-17. https://doi.org/10.1016/j.visinf.2022.10.001
38. Martínez, F., Barraza, C., Gonzalez, N., & González, J. (2016). KAPEAN: Understanding Affective States of Children with ADHD. Educational Technology & Society, 19(2), 18–28. Retrieved from https://www.jstor.org/stable/pdf/jeductechsoci.19.2.18.pdf
39. Mayer, R. E. (2020). Searching for the role of emotions in e-learning. Learning and Instruction, 70, 101213. https://doi.org/10.1016/j.learninstruc.2019.05.010
40. Muhammad, G., & Hossain, M. S. (2021). Emotion recognition for cognitive edge computing using deep learning. IEEE Internet of Things Journal, 8(23), 16894-16901.
41. Nicolaou, M. A., Gunes, H., & Pantic, M. (2012). Output-associative RVM regression for dimensional and continuous emotion prediction. Image and Vision Computing, 30(3), 186-196. https://doi.org/10.1016/j.imavis.2011.12.005
42. Obergriesser, S., & Stoeger, H. (2020). Students’ emotions of enjoyment and boredom and their use of cognitive learning strategies–How do they affect one another?. Learning and Instruction, 66, 101285. https://doi.org/10.1016/j.learninstruc.2019.101285
43. Öztüre, G., Fidan, A., Bakır, E., Uslu, N. A., & Usluel, Y. (2021). Eğitsel bağlamda teknoloji ve duygu çalışmaları üzerine bir sistematik haritalama çalışması: tanımlar, kuramlar ve gelecekteki yönelimler [A systematic mapping study on technology and emotion studies in educational contexts: definitions, theories and future directions.]. Eğitim Teknolojisi Kuram ve Uygulama, 11(1), 20-47. https://doi.org/10.17943/etku.745236
44. Peng, X., & Xu, Q. (2020). Investigating learners' behaviors and discourse content in MOOC course reviews. Computers & Education, 143, 103673. https://doi.org/10.1016/j.compedu.2019.103673
45. Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of educational Psychology, 95(4), 667. https://doi.org/10.1037/0022-0663.95.4.667
46. Putwain, D. W., Becker, S., Symes, W., & Pekrun, R. (2018). Reciprocal relations between students’ academic enjoyment, boredom, and achievement over time. Learning and Instruction, 54, 73-81. https://doi.org/10.1016/j.learninstruc.2017.08.004
47. Said, Y., & Barr, M. (2021). Human emotion recognition based on facial expressions via deep learning on high-resolution images. Multimedia Tools and Applications, 80(16), 25241-25253. https://doi.org/10.1007/s11042-021-10918-9
48. Sakalle, A., Tomar, P., Bhardwaj, H., Acharya, D., & Bhardwaj, A. (2021). A LSTM based deep learning network for recognizing emotions using wireless brainwave driven system. Expert Systems with Applications, 173, 114516. https://doi.org/10.1016/j.eswa.2020.114516
49. Sarrafzadeh, A., Alexander, S., Dadgostar, F., Fan, C., & Bigdeli, A. (2008). “How do you know that I don’t understand?” A look at the future of intelligent tutoring systems. Computers in Human Behavior, 24(4), 1342-1363. https://doi.org/10.1016/j.chb.2007.07.008
50. Savci, P., & Das, B. (2023). Prediction of the customers’ interests using sentiment analysis in e-commerce data for comparison of Arabic, English, and Turkish languages. Journal of King Saud University - Computer and Information Sciences, 35(3), 227–237. https://doi.org/10.1016/j.jksuci.2023.02.017
51. Scherer, K. R. (2005). What are emotions? And how can they be measured?. Social science information, 44(4), 695-729. https://doi.org/10.1177/0539018405058216
52. Schneider, S., Krieglstein, F., Beege, M., & Rey, G. D. (2022). The impact of video lecturers’ nonverbal communication on learning–An experiment on gestures and facial expressions of pedagogical agents. Computers & Education, 176, 104350. https://doi.org/10.1016/j.compedu.2021.104350
53. Sealy, M. K. (2021). Communication in the Time of COVID-19: An examination of imagined interactions and communication apprehension during the COVID-19 Pandemic. Imagination, Cognition and Personality, 41(2), 158-186. https://doi.org/10.1177/02762366211021076
54. Sethi, K., & Jaiswal, V. (2022). PSU-CNN: prediction of student understanding in the classroom through student facial images using convolutional neural network. Materials Today: Proceedings, 62, 4957-4964. https://doi.org/10.1016/j.matpr.2022.03.691
55. Sydänmaanlakka, A., Häsä, J., Holm, M. E., & Hannula, M. S. (2024). Mathematics-related achievement emotions – Interaction between learning environment and students’ mathematics performance. Learning and Individual Differences, 113, 102486. https://doi.org/10.1016/j.lindif.2024.102486
56. Taub, M., Sawyer, R., Smith, A., Rowe, J., Azevedo, R., & Lester, J. (2020). The agency effect: The impact of student agency on learning, emotions, and problem-solving behaviors in a game-based learning environment. Computers & Education, 147, 103781. https://doi.org/10.1016/j.compedu.2019.103781
57. Tonguç, G., & Ozkara, B. O. (2020). Automatic recognition of student emotions from facial expressions during a lecture. Computers & Education, 148, 103797. https://doi.org/10.1016/j.compedu.2019.103797
58. Uçar, H. (2017). Açık ve uzaktan öğrenmede motivasyon tasarımı [Motivation design in open and distance learning]. V. Yüzer (Eds.) Açık ve Uzaktan Öğrenmede Bireysel Farklılıklar [Individual Differences in Open and Distance Learning] (pp.47-69). Anadolu Üniversitesi Açıköğretim Fakültesi Yayını.
59. Wang, W., Xu, K., Niu, H., & Miao, X. (2020). [Retracted] Emotion Recognition of Students Based on Facial Expressions in Online Education Based on the Perspective of Computer Simulation. Complexity, 2020(1), 4065207. https://doi.org/10.1155/2020/4065207
60. Yadegaridehkordi, E., Noor, N. F. B. M., Ayub, M. N. B., Affal, H. B., & Hussin, N. B. (2019). Affective computing in education: A systematic review and future research. Computers & Education, 142, 103649. https://doi.org/10.1016/j.compedu.2019.103649
61. Yolcu, H. H. (2020). Koronavirüs (covid-19) pandemi sürecinde sınıf öğretmeni adaylarının uzaktan eğitim deneyimleri [Distance education experiences of classroom teacher candidates during the coronavirus (covid-19) pandemic]. Açıköğretim Uygulamaları ve Araştırmaları Dergisi, 6(4), 237-250. Retrieved from https://dergipark.org.tr/en/pub/auad/issue/57638/788890
62. Yoshitomi, Y., Kim, S., Kawano, T., & Kilazoe, T. (2000). Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In 9th IEEE International Workshop on Robot and Human Interactive Communication, pp. 178–183.
63. Yuan, Y., Li, Y., Nie, J., Chao, X., & Li, J. (2024). An individual adaptive ubiquitous learning paradigm: Focusing on the collection and utilization of academic emotions. Computers in Human Behavior, 156, 108228. https://doi.org/10.1016/j.chb.2024.108228
64. Zembylas, M., Theodorou, M., & Pavlakis, A. (2008). The role of emotions in the experience of online learning: Challenges and opportunities. Educational Media International, 45(2), 107-117. https://doi.org/10.1080/09523980802107237