Facial Expression Analysis in Computer-Assisted Learning Environments During Reading and Problem-Solving Processes

Adem Mehmet Yıldız; Ömer Kırmacı

doi:10.14686/buefad.1765379

Facial Expression Analysis in Computer-Assisted Learning Environments During Reading and Problem-Solving Processes

Abstract

This study aims to analyze students’ facial expressions during reading and problem-solving tasks in a computer-assisted learning environment, focusing on their relationship with gender, academic performance, and time. Due to its exploratory nature, the research was conducted using a single-group post-test design. The study group consisted of 40 university students (23 female and 17 male). Participants were asked to read a scientific text and then solve a multiple-choice test. During the task, students’ facial expressions were recorded through the computer’s webcam and analyzed in real-time by a facial expression recognition algorithm measuring valence, arousal, and intensity. The collected data were analyzed using t-tests and regression analysis. Female students exhibited lower valence and higher intensity in their facial expressions compared to male students. No significant relationship was found between facial expressions and academic performance or reading duration. However, valence levels were observed to decrease as problem-solving time increased. These findings contribute to the literature on emotional state analysis in educational settings.

Keywords

References

Ahn, B. “Tony”, & Harley, J. M. (2020). Facial expressions when learning with a Queer History App: Application of the Control Value Theory of Achievement Emotions. British Journal of Educational Technology, 51(5), 1563–1576. https://doi.org/10.1111/BJET.12989
Ainley, M., Corrigan, M., & Richardson, N. (2005). Students, tasks and emotions: Identifying the contribution of emotions to students’ reading of popular culture and popular science texts. Learning and Instruction, 15(5), 433–447. https://doi.org/10.1016/J.LEARNINSTRUC.2005.07.011
Ainley, M., Hillman, K., & Hidi, S. (2002). Gender and interest processes in response to literary texts: Situational and individual interest. Learning and Instruction, 12(4), 411–428. https://doi.org/10.1016/S0959-4752(01)00008-1
Artino, A. R. (2009a). Online learning: Are subjective perceptions of instructional context related to academic success? The Internet and Higher Education, 12(3–4), 117–125. https://doi.org/10.1016/J.IHEDUC.2009.07.003
Artino, A. R. (2009b). Think, feel, act: Motivational and emotional influences on military students’ online academic success. Journal of Computing in Higher Education, 21(2), 146–166. https://doi.org/10.1007/S12528-009-9020-9/TABLES/2
Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements. Psychological Science in the Public Interest, 20(1), 1–68. https://doi.org/10.1177/1529100619832930
Brody, L. R., & Hall, J. A. (2008). Gender and Emotion in Context. In Handbook of Emotions (3rd edn, pp. 395–409). THE GUILFORD PRESS.
Cheng, S., Hu, Y., Fan, J., & Wei, Q. (2020). Reading comprehension based on visualization of eye tracking and EEG data. Science China. Information Sciences, 63(11), 214101. https://doi.org/10.1007/s11432-019-1466-7

Chickerur, S., & Joshi, K. (2015). 3D face model dataset: Automatic detection of facial expressions and emotions for educational environments. British Journal of Educational Technology, 46(5), 1028–1037. https://doi.org/10.1111/BJET.12325
Creswell, J. W., & Creswell, J. D. (2017). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. SAGE Publications.
Daley, S. G., Willett, J. B., & Fischer, K. W. (2014). Emotional responses during reading: Physiological responses predict real-time reading comprehension. Journal of Educational Psychology, 106(1), 132–143. https://doi.org/10.1037/a0033408
DeVellis, R. F. (2012). Scale Development: Theory and Applications. SAGE.
Dever, D. A., Wiedbusch, M. D., Cloude, E. B., Lester, J., & Azevedo, R. (2022). Emotions and the Comprehension of Single versus Multiple Texts during Game-based Learning. Discourse Processes, 59(1–2), 94–115. https://doi.org/10.1080/0163853X.2021.1950450
Dindar, M. (2015). Teknoloji Destekli Öğrenme Ortamlarında Çoklu Görev Yapmanın Öğrenmeye Etkisi [PhD Thesis]. https://www.proquest.com/docview/2568540335/abstract/589A64977FCB40FEPQ/1
Dindar, M., & Akbulut, Y. (2016). Effects of multitasking on retention and topic interest. Learning and Instruction, 41, 94–105. https://doi.org/10.1016/J.LEARNINSTRUC.2015.10.005
D’Mello, S. (2013). A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. Journal of Educational Psychology, 105(4), 1082–1099. https://doi.org/10.1037/A0032674
D’Mello, S. K., & Graesser, A. (2010). Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modeling and User-Adapted Interaction, 20(2), 147–187. https://doi.org/10.1007/s11257-010-9074-4
Dönmez, O., & Akbulut, Y. (2021). Timing and relevance of secondary tasks impact multitasking performance. Computers & Education, 161, 104078. https://doi.org/10.1016/J.COMPEDU.2020.104078
Ekman, P. (1993). Facial expression and emotion. American Psychologist, 48(4), 384–392. https://doi.org/10.1037/0003-066X.48.4.384
Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2), 124–129. https://doi.org/10.1037/h0030377
Ekman, P., & Friesen, W. V. (1978). Facial Action Coding System Investigator’s Guide. Consulting Psychologists Press.
Ekornes, S. (2022). The impact of perceived psychosocial environment and academic emotions on higher education students’ intentions to drop out. Higher Education Research & Development, 41(4), 1044–1059. https://doi.org/10.1080/07294360.2021.1882404
Fida, A., Umer, M., Saidani, O., Hamdi, M., Alnowaiser, K., Bisogni, C., Abate, A. F., & Ashraf, I. (2023). Real time emotions recognition through facial expressions. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-023-16722-x
Gao, W., Cao, B., Shan, S., Chen, X., Zhou, D., Zhang, X., & Zhao, D. (2008). The CAS-PEAL Large-Scale Chinese Face Database and Baseline Evaluations. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 38(1), 149–161. https://doi.org/10.1109/TSMCA.2007.909557
Goeleven, E., Raedt, R. D., Leyman, L., & Verschuere, B. (2008). The Karolinska Directed Emotional Faces: A validation study. Cognition and Emotion, 22(6), 1094–1118. https://doi.org/10.1080/02699930701626582
Graesser, A. C., & D’Mello, S. (2012). Moment-To-Moment Emotions During Reading. The Reading Teacher, 66(3), 238–242. https://doi.org/10.1002/TRTR.01121
Grafsgaard, J. F., Wiggins, J. B., Boyer, K. E., Wiebe, E. N., & Lester, J. C. (2013). Automatically Recognizing Facial Expression: Predicting Engagement and Frustration.
Gray, D. E. (2004). Doing research in the real world. Sage Publications Ltd. https://openresearch.surrey.ac.uk/esploro/outputs/book/Doing-research-in-the-real-world/99516972502346
Han, L., Osman, S. Z. M., & Che, J. (2024). Impact of Online Learning Interactive Experience on Language Learners’ Emotional Engagement. Eurasian Journal of Applied Linguistics, 10(1). https://www.proquest.com/docview/3234775195/abstract/36EA28451B164FA4PQ/1
Hascher, T. (2010). Learning and Emotion: Perspectives for Theory and Research. European Educational Research Journal, 9(1), 13–28. https://doi.org/10.2304/eerj.2010.9.1.13
HHMI. (2013a). Malaria: Human Host. HHMI BioInteractive. https://www.biointeractive.org/sites/default/files/media/file/2022-04/MalariaHumanHost-Transcript-anim.pdf
HHMI. (2013b). Malaria: Mosquito Host. HHMI BioInteractive. https://www.biointeractive.org/sites/default/files/media/file/2022-04/MalariaMosquitoHost-Transcript-anim.pdf
Hopwood, N., Palmer, T.-A., Koh, G. A., Lai, M. Y., Dong, Y., Loch, S., & Yu, K. (2024). Understanding student emotions when completing assessment: Technological, teacher and student perspectives. INTERNATIONAL JOURNAL OF RESEARCH & METHOD IN EDUCATION. https://doi.org/10.1080/1743727X.2024.2358792
Jena, R. K. (2019). Understanding academic achievement emotions towards business analytics course: A case study among business management students from India. Computers in Human Behavior, 92, 716–723. https://doi.org/10.1016/J.CHB.2018.08.024
Kollias, D., & Zafeiriou, S. (2021, March). Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework. In arXiv.org. https://arxiv.org/abs/2103.15792v1
Kumar, J. A. (2024). Facial animacy in anthropomorphised designs: {Insights} from leveraging self-report and facial expression analysis for multimedia learning. Computers & Education, 223, 105150. https://doi.org/10.1016/j.compedu.2024.105150
Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H. J., Hawk, S. T., & Knippenberg, A. van. (2010). Presentation and validation of the Radboud Faces Database. Cognition and Emotion, 24(8), 1377–1388. https://doi.org/10.1080/02699930903485076
Laskay-Horváth, C., Aranyi, G., Pachner, O., Remete, E. P., & Kemény, F. (2025). How Is Reading Related to Working Memory? —A Cross-Sectional Developmental Perspective. Reading Research Quarterly, 60(3), e70019. https://doi.org/10.1002/rrq.70019
Martucci, A., Graziani, D., Bei, E., Bischetti, L., Bambini, V., Gursesli, M. C., Guazzini, A., & Pecini, C. (2025). Emotional engagement in a humor-understanding reading task: An AI study perspective. Current Psychology, 44(9), 7818–7831. https://doi.org/10.1007/s12144-024-07228-2
Mason, L., Scrimin, S., Zaccoletti, S., Tornatora, M. C., & Goetz, T. (2018). Webpage reading: Psychophysiological correlates of emotional arousal and regulation predict multiple-text comprehension. Computers in Human Behavior, 87, 317–326. https://doi.org/10.1016/j.chb.2018.05.020
Mercimek, B., Akbulut, Y., Dönmez, O., & Sak, U. (2020). Multitasking impairs learning from multimedia across gifted and non-gifted students. Educational Technology Research and Development, 68(3), 995–1016. https://doi.org/10.1007/S11423-019-09717-9/TABLES/3
Mishra, P., Pandey, C. M., Singh, U., Gupta, A., Sahu, C., & Keshri, A. (2019). Descriptive statistics and normality tests for statistical data. Annals of Cardiac Anaesthesia, 22(1), 67–72. https://doi.org/10.4103/aca.ACA_157_18
Moreno, R., & Mayer, R. (2007). Interactive Multimodal Learning Environments. Educational Psychology Review, 19(3), 309–326. https://doi.org/10.1007/s10648-007-9047-2
Niculescu, A. C., Tempelaar, D., Leppink, J., Dailey-Hebert, A., Segers, M., & Gijselaers, W. (2015). Feelings and performance in the first year at university: Learning-related emotions as predictors of achievement outcomes in mathematics and statistics. Electronic Journal of Research in Education Psychology, 13(37), 431–462. https://doi.org/10.25115/EJREP.37.15012
Olszanowski, M., Pochwatko, G., Kuklinski, K., Scibor-Rylski, M., Lewinski, P., & Ohme, R. K. (2015). Warsaw set of emotional facial expression pictures: A validation study of facial display photographs. Frontiers in Psychology, 5. https://www.frontiersin.org/articles/10.3389/fpsyg.2014.01516
Örün, Ö., & Akbulut, Y. (2019). Effect of multitasking, physical environment and electroencephalography use on cognitive load and retention. Computers in Human Behavior, 92, 216–229. https://doi.org/10.1016/J.CHB.2018.11.027
Patrick, F. (2023, July). Yüz algılama ve öznitelikler—Yüz—Azure AI services. https://learn.microsoft.com/tr-tr/azure/ai-services/computer-vision/concept-face-detection
Pekrun, R. (2006). The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice. Educational Psychology Review, 18(4), 315–341. https://doi.org/10.1007/S10648-006-9029-9/FIGURES/2
Pekrun, R. (2011). Emotions as Drivers of Learning and Cognitive Development. New Perspectives on Affect and Learning Technologies, 23–39. https://doi.org/10.1007/978-1-4419-9625-1_3
Pekrun, R. (2022). Emotions in Reading and Learning from Texts: Progress and Open Problems. Discourse Processes, 59(1–2), 116–125. https://doi.org/10.1080/0163853X.2021.1938878
Pekrun, R., Götz, T., Titz, W., & Perry, R. P. (2002). Positive emotions in education. http://kops.uni-konstanz.de/handle/123456789/13908
Pekrun, R., & Linnenbrink-Garcia, L. (2014). International handbook of emotions in education. International Handbook of Emotions in Education, 1–698. https://doi.org/10.4324/9780203148211/INTERNATIONAL-HANDBOOK-EMOTIONS-EDUCATION-REINHARD-PEKRUN-LISA-LINNENBRINK-GARCIA/ACCESSIBILITY-INFORMATION
Pekrun, R., Vogl, E., Muis, K. R., & Sinatra, G. M. (2017). Measuring emotions during epistemic activities: The Epistemically-Related Emotion Scales. Cognition and Emotion, 31(6), 1268–1276. https://doi.org/10.1080/02699931.2016.1204989
Prayogo, A., Khotimah, K., Istiqomah, L., & Maharsi, I. (2024). Students’ emotional engagement in online classes: A conceptual framework. International Journal of Information and Learning Technology, 41(1), 61–72. https://doi.org/10.1108/IJILT-04-2023-0052
Rasouli, A., Kotseruba, I., Kunic, T., & Tsotsos, J. K. (2019). PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction (pp. 6262–6271). http://data.nvision2.
Rucks-Ahidiana, Z. (2014). Navigating the Lack of Face Time: The Instructor Role in the Online Classroom. In CASES ON CRITICAL AND QUALITATIVE PERSPECTIVES IN ONLINE HIGHER EDUCATION (pp. 98–116). https://doi.org/10.4018/978-1-4666-5051-0.ch006
Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178. https://doi.org/10.1037/h0077714
Ruthig, J. C., Perry, R. P., Hladkyj, S., Hall, N. C., Pekrun, R., & Chipperfield, J. G. (2008). Perceived control and emotions: Interactive effects on performance in achievement settings. Social Psychology of Education, 11(2), 161–180. https://doi.org/10.1007/S11218-007-9040-0/METRICS
Scherer, K. R., & Ekman, P. (1982). Handbook of methods in nonverbal behavior research. Cambridge University Press. https://cir.nii.ac.jp/crid/1130000796865038208
Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8(4), 350–353. https://doi.org/10.1037/1040-3590.8.4.350
Sharma, K., Giannakos, M., & Dillenbourg, P. (2020). Eye-tracking and artificial intelligence to enhance motivation and learning. Smart Learning Environments, 7(1), 13. https://doi.org/10.1186/s40561-020-00122-x
Stolba, A., Hope, A., Branch, J., Manoj, P., Trinier, J., Behboudi, A., VanOostveen, R., & Childs, E. (2024). Beyond content delivery: Harnessing emotional intelligence for community building in fully online digital spaces. INTERNATIONAL JOURNAL OF EDUCATIONAL TECHNOLOGY IN HIGHER EDUCATION, 21(1). https://doi.org/10.1186/s41239-024-00481-2
Tabachnick, B. G., & Fidell, L. S. (2006). Using multivariate statistics .
Turan, C., Neergaard, K. D., & Lam, K. M. (2022). Facial Expressions of Comprehension (FEC). IEEE Transactions on Affective Computing, 13(1), 335–346. https://doi.org/10.1109/TAFFC.2019.2954498
Turan, C., Wang, Y., Lai, S.-C., Neergaard, K. D., & Lam, K.-M. (2018). Facial Expressions of Sentence Comprehension. 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP), 1–5. https://doi.org/10.1109/ICDSP.2018.8631633
Vonikakis, V., & Winkler, S. (2020). Identity-Invariant Facial Landmark Frontalization For Facial Expression Analysis. 2020 IEEE International Conference on Image Processing (ICIP), 2281–2285. https://doi.org/10.1109/ICIP40778.2020.9190989
Vonikakis, V., & Winkler, S. (2021). Efficient Facial Expression Analysis For Dimensional Affect Recognition Using Geometric Features. http://arxiv.org/abs/2106.07817
Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: {The} {PANAS} scales. Journal of Personality and Social Psychology, 54(6), 1063–1070. https://doi.org/10.1037/0022-3514.54.6.1063
Xiao, J. (2017). Learner-content interaction in distance education: The weakest link in interaction research. Distance Education. (world). https://www.tandfonline.com/doi/abs/10.1080/01587919.2017.1298982
Xu, X., Dugdale, D. M., Wei, X., & Mi, W. (2023). Leveraging Artificial Intelligence to Predict Young Learner Online Learning Engagement. AMERICAN JOURNAL OF DISTANCE EDUCATION, 37(3), 185–198. https://doi.org/10.1080/08923647.2022.2044663
You, J. W., & Kang, M. (2014). The role of academic emotions in the relationship between perceived academic control and self-regulated learning in online learning. Computers & Education, 77, 125–133. https://doi.org/10.1016/J.COMPEDU.2014.04.018
Zhang, C., Wang, Z., Fang, Z., & Xiao, X. (2024). Guiding student learning in video lectures: Effects of instructors’ emotional expressions and visual cues. COMPUTERS & EDUCATION, 218. https://doi.org/10.1016/j.compedu.2024.105062
Zhang, H., & Wang, M. (2024). A novel deep learning model to improve the recognition of students’ facial expressions in online learning environments. KNOWLEDGE MANAGEMENT & E-LEARNING-AN INTERNATIONAL JOURNAL, 16(1). https://doi.org/10.34105/j.kmel.2024.16.006.

Details

Primary Language

English

Subjects

Instructional Technologies

Journal Section

Research Article

Authors

Adem Mehmet Yıldız ^*
0000-0002-4033-0122
Türkiye

Ömer Kırmacı
0000-0002-0954-1263
Türkiye

Publication Date

February 3, 2026

Submission Date

August 15, 2025

Acceptance Date

December 30, 2025

Published in Issue

Year 2026 Volume: 15 Number: 1

DOI

https://doi.org/10.14686/buefad.1765379

IZ

https://izlik.org/JA99AA56UP

Cite

RIS / Bibtex

APA

Yıldız, A. M., & Kırmacı, Ö. (2026). Facial Expression Analysis in Computer-Assisted Learning Environments During Reading and Problem-Solving Processes. Bartın University Journal of Faculty of Education, 15(1), 287-305. https://doi.org/10.14686/buefad.1765379

AMA

1.Yıldız AM, Kırmacı Ö. Facial Expression Analysis in Computer-Assisted Learning Environments During Reading and Problem-Solving Processes. BUEFAD. 2026;15(1):287-305. doi:10.14686/buefad.1765379

Chicago

Yıldız, Adem Mehmet, and Ömer Kırmacı. 2026. “Facial Expression Analysis in Computer-Assisted Learning Environments During Reading and Problem-Solving Processes”. Bartın University Journal of Faculty of Education 15 (1): 287-305. https://doi.org/10.14686/buefad.1765379.

EndNote

Yıldız AM, Kırmacı Ö (February 1, 2026) Facial Expression Analysis in Computer-Assisted Learning Environments During Reading and Problem-Solving Processes. Bartın University Journal of Faculty of Education 15 1 287–305.

IEEE

[1]A. M. Yıldız and Ö. Kırmacı, “Facial Expression Analysis in Computer-Assisted Learning Environments During Reading and Problem-Solving Processes”, BUEFAD, vol. 15, no. 1, pp. 287–305, Feb. 2026, doi: 10.14686/buefad.1765379.

ISNAD

Yıldız, Adem Mehmet - Kırmacı, Ömer. “Facial Expression Analysis in Computer-Assisted Learning Environments During Reading and Problem-Solving Processes”. Bartın University Journal of Faculty of Education 15/1 (February 1, 2026): 287-305. https://doi.org/10.14686/buefad.1765379.

JAMA

1.Yıldız AM, Kırmacı Ö. Facial Expression Analysis in Computer-Assisted Learning Environments During Reading and Problem-Solving Processes. BUEFAD. 2026;15:287–305.

MLA

Yıldız, Adem Mehmet, and Ömer Kırmacı. “Facial Expression Analysis in Computer-Assisted Learning Environments During Reading and Problem-Solving Processes”. Bartın University Journal of Faculty of Education, vol. 15, no. 1, Feb. 2026, pp. 287-05, doi:10.14686/buefad.1765379.

Vancouver

1.Adem Mehmet Yıldız, Ömer Kırmacı. Facial Expression Analysis in Computer-Assisted Learning Environments During Reading and Problem-Solving Processes. BUEFAD. 2026 Feb. 1;15(1):287-305. doi:10.14686/buefad.1765379