DUYGUSAL TASARIMLI PEDAGOJİK AJANLARIN SOSYAL ARACILIK ETKİSİ: MULTİMEDYA ÖĞRENMEDE YÜZ İFADESİ ANALİZİ

Öz

Bu çalışma, duygusal yüz ifadelerine sahip pedagojik ajanların çoklu ortam öğrenme süreçlerinde öğrencilerin yüz ifadelerine yansıyan duygusal tepkiler üzerindeki etkisini incelemektedir. Araştırmanın amacı, yüz ifadesi analizini eğitim teknolojilerine entegre ederek öğrenme sürecindeki duygusal katılımı nesnel biçimde ölçmek ve farklı duygusal tasarımlara sahip pedagojik ajanların öğrenenlerin duygusal ve bilişsel süreçleri üzerindeki etkilerini ortaya koymaktır. Çalışma, 2024–2025 akademik yılında Türkiye’de bir devlet üniversitesinde yürütülmüş ve iki farklı çoklu ortam uygulaması (“IP ve IP Sınıfları” ve “Sıtmanın Yaşam Döngüsü”) kapsamında toplam 162 lisans öğrencisiyle gerçekleştirilmiştir. Katılımcılar üç gruba ayrılmıştır: yalnızca metin ve sesli geri bildirim alan kontrol grubu, nötr yüz ifadesine sahip pedagojik ajanla etkileşime giren deney 1 grubu ve neşeli yüz ifadesine sahip ajanla etkileşime giren deney 2 grubu. Python tabanlı bir yüz ifadesi analiz algoritmasıyla öğrencilerin yüz ifadelerinden değerlik, uyarılma ve yoğunluk değerleri saniyede dört kez ölçülmüş ve üç öğrenme aşamasında (videoyu izleme, gömülü soruları yanıtlama ve geri bildirimleri alma) analiz edilmiştir. ANOVA sonuçları her iki uygulamada da gruplar arasında özellikle değerlik parametresinde anlamlı farklılıklar olduğunu göstermiştir. Deney gruplarındaki öğrencilerin yüz ifadelerinde kontrol grubuna göre daha negatif değerlik gözlenmiştir. Bu durum, pedagojik ajanların sosyal ipuçlarının öğrencilerde daha yüksek dikkat, bilişsel çaba ve duygusal uyarılma oluşturduğunu göstermektedir. Ek olarak gömülü sorulara verilen doğru ya da yanlış yanıtlar öğrencilerin yüz ifadeleri üzerinde etkili olmamıştır. Bulgular, Sosyal Aracılık Teorisi, Duygusal Tepki Teorisi ve Tekinsiz Vadi hipotezi çerçevesinde yorumlanmıştır. Sonuç olarak, duygusal tasarımlı pedagojik ajanların öğrenme sürecinde mikro yüz ifadeleri aracılığıyla ölçülebilen anlamlı bilişsel ve duyuşsal farklılıklar yarattığı, yüz ifadesi analizinin ise eğitimde duygusal etkileşimi değerlendirmede güçlü bir araç olduğu ortaya konmuştur.

Anahtar Kelimeler

• pedagojik ajanlar
• yüz ifadesi analizi
• çoklu ortamla öğrenme
• duygusal tasarım
• değerlik
• uyarılma
• sosyal aracılık teorisi

Etik Beyan

Yasal İzinler: Kırklareli Üniversitesi Bilimsel Araştırma ve Yayın Etiği Kurulu, Tarih: 18.11.2024, Sayı: E-35523585-199-141965.

SOCIAL AGENCY EFFECTS of EMOTIONALLY DESIGNED AGENTS: FACIAL EXPRESSION ANALYSIS IN MULTIMEDIA LEARNING

Öz

This study investigates the effects of pedagogical agents with emotional facial expressions on learners’ affective responses as reflected in their own facial expressions during multimedia learning processes. The aim of the research is to integrate facial expression analysis into educational technologies in order to objectively measure emotional engagement throughout the learning process and to reveal how pedagogical agents with different emotional designs influence learners’ emotional and cognitive processes. The study was conducted at a public university in Türkiye during the 2024–2025 academic year and involved a total of 162 undergraduate students across two multimedia learning applications (“IP and IP Classes” and “The Life Cycle of Malaria”). Participants were divided into three groups: a control group receiving feedback only through text and audio, an experimental group interacting with a pedagogical agent displaying a neutral facial expression, and another experimental group interacting with a cheerful agent. Using a Python-based facial expression analysis algorithm, students’ facial expressions were continuously measured to capture valence, arousal, and intensity values four times per second and analyzed across three learning phases: watching the video, answering embedded questions, and receiving feedback. The ANOVA results indicated that in both multimedia applications there were statistically significant differences among the groups, particularly in the valence parameter. Students in the experimental groups exhibited more negative valence compared to those in the control group. This finding suggests that the social cues conveyed by pedagogical agents induced greater attention, cognitive effort, and emotional arousal among learners. Furthermore, whether the responses to embedded questions were correct or incorrect had no significant impact on students' facial expressions. The results were interpreted within the frameworks of Social Agency Theory, Emotional Response Theory, and the Uncanny Valley hypothesis. In conclusion, pedagogical agents with emotional facial designs were found to generate meaningful cognitive and affective differences during learning, observable through micro-level facial expressions, while facial expression analysis proved to be a powerful tool for evaluating emotional engagement in educational settings.

Anahtar Kelimeler

• pedagogical agents
• facial expression analysis
• multimedia learning
• emotional design
• valence
• arousal
• social agency theory

Etik Beyan

Legal Permissions: Kırklareli University Scientific Research and Publication Ethics Committee, Date: 18.11.2024, Number: E-35523585-199-141965.

Kaynakça

1. Ayvaz, U., Gürüler, H., ve Devrim, M. O. (2017). USE OF FACIAL EMOTION RECOGNITION IN E-LEARNING SYSTEMS. https://doi.org/10.33407/itlt.v60i4.1743
2. Baddeley, A. (1992). Working Memory. Science. https://doi.org/10.1126/science.1736359
3. Brom, C., Stárková, T., ve D’Mello, S. K. (2018). How effective is emotional design? A meta-analysis on facial anthropomorphisms and pleasant colors during multimedia learning. Educational Research Review, 25, 100-119. https://doi.org/10.1016/j.edurev.2018.09.004
4. Büyüköztürk, Ş., Kılıç Çakmak, E., Akgün, Ö., Karadeniz Oran, Ş., ve Demirel, F. (2019). Bilimsel Araştırma Yöntemleri. Pegem Akademi. https://avesis.gazi.edu.tr/yayin/491a4be9-de92-4456-ab17-eff747c88774/bilimsel-arastirma-yontemleri
5. Chen, G.-D., Lee, J.-H., Wang, C.-Y., Chao, P.-Y., Li, L.-Y., ve Lee, T.-Y. (2012). An Empathic Avatar in a Computer-Aided Learning Program to Encourage and Persuade Learners. Journal of Educational Technology & Society, 15(2), 62-72.
6. Creswell, J. W., ve Creswell, J. D. (2017). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. SAGE Publications.
7. Deng, R., ve Gao, Y. (2023). Effects of embedded questions in pre-class videos on learner perceptions, video engagement, and learning performance in flipped classrooms. Active Learning in Higher Education, 14697874231167098. https://doi.org/10.1177/14697874231167098
8. D’Mello, S. K., ve Graesser, A. (2010). Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modeling and User-Adapted Interaction, 20(2), 147-187. https://doi.org/10.1007/s11257-010-9074-4

9. Ekman, P. (1993). Facial expression and emotion. American Psychologist, 48(4), 384-392. https://doi.org/10.1037/0003-066X.48.4.384
10. Ekman, P. (2004). Emotions revealed. BMJ, 328(Suppl S5), 0405184. https://doi.org/10.1136/sbmj.0405184
11. Ekman, P., ve Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2), 124-129. https://doi.org/10.1037/h0030377
12. Ekman, P., ve Friesen, W. V. (1978). Facial Action Coding System Investigator’s Guide. Consulting Psychologists Press.
13. Fiedler, K., ve Beier, S. (2014). Affect and cognitive processes in educational contexts. Içinde International Handbook of Emotions in Education (ss. 36-55). Scopus. https://doi.org/10.4324/9780203148211
14. Frechette, C., ve Moreno, R. (2010). The Roles of Animated Pedagogical Agents’ Presence and Nonverbal Communication in Multimedia Learning Environments. Journal of Media Psychology, 22(2), 61-72. https://doi.org/10.1027/1864-1105/a000009
15. Gao, W., Cao, B., Shan, S., Chen, X., Zhou, D., Zhang, X., ve Zhao, D. (2008). The CAS-PEAL Large-Scale Chinese Face Database and Baseline Evaluations. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 38(1), 149-161. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans. https://doi.org/10.1109/TSMCA.2007.909557
16. Goeleven, E., De Raedt, R., Leyman, L., ve Verschuere, B. (2008). The Karolinska Directed Emotional Faces: A validation study. Cognition and Emotion, 22(6), 1094-1118. https://doi.org/10.1080/02699930701626582
17. Gomes, C. F. A., Brainerd, C. J., ve Stein, L. M. (2013). Effects of emotional valence and arousal on recollective and nonrecollective recall. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39(3), 663-677. https://doi.org/10.1037/a0028578
18. Hattie, J., ve Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112. https://doi.org/10.3102/003465430298487
19. Horan, S. M., Martin, M. M., ve Weber, K. (2012). Understanding Emotional Response Theory: The Role of Instructor Power and Justice Messages. Communication Quarterly, 60(2), 210-233. https://doi.org/10.1080/01463373.2012.669323
20. Hung, S.-T. A. (2016). Enhancing feedback provision through multimodal video technology. Computers & Education, 98, 90-101. https://doi.org/10.1016/j.compedu.2016.03.009
21. İzmirli, S., Kırmacı, Ö., ve Kahraman, A. (2017). Eğitsel Ajan Araştırmalarında Güncel Eğilimler: 2009-2014 Yılları Arasındaki Makalelerin İçerik Analizi. Ege Eğitim Dergisi, 18(1), 213-243. https://doi.org/10.12984/egeefd.328382
22. Kestin, G., ve Miller, K. (2022). Harnessing active engagement in educational videos: Enhanced visuals and embedded questions. Physical Review Physics Education Research, 18(1), 010148. https://doi.org/10.1103/PhysRevPhysEducRes.18.010148
23. Kollias, D., ve Zafeiriou, S. (2021, Mart 29). Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework. arXiv.Org. https://arxiv.org/abs/2103.15792v1
24. Landau, T. (1989). About Faces. Doubleday Anchor Books.
25. Lang, Y., Gong, S., Hu, X., Xiao, B., Wang, Y., Jiang, T., ve Guo, J. (2024). The impact of pedagogical agent’s emotional expression on multimedia learning: a multi-faceted perspective. Journal of Computing in Higher Education. https://doi.org/10.1007/s12528-024-09429-x
26. Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H. J., Hawk, S. T., ve van Knippenberg, A. (2010). Presentation and validation of the Radboud Faces Database. Cognition and Emotion, 24(8), 1377-1388. https://doi.org/10.1080/02699930903485076
27. Lavater, J. C. (1775). Physiognomische Fragmente, zur Beförderung der Menschenkenntniß und Menschenliebe (1. Auflage). Weidmann und Reich; Steiner.
28. Lester, J. C., Towns, S. G., Callaway, C. B., Voerman, J. L., ve FitzGerald, P. J. (2000). Deictic and Emotive Communication in Animated Pedagogical Agents. https://doi.org/10.7551/mitpress/2697.003.0007
29. Liew, T. W., Mat Zin, N. A., ve Sahari, N. (2017). Exploring the affective, motivational and cognitive effects of pedagogical agent enthusiasm in a multimedia learning environment. Human-Centric Computing and Information Sciences, 7(1), 9. https://doi.org/10.1186/s13673-017-0089-2
30. Liew, T. W., Zin, N. A. M., Sahari, N., ve Tan, S.-M. (2016). The Effects of a Pedagogical Agent’s Smiling Expression on the Learner’s Emotions and Motivation in a Virtual Learning Environment. The International Review of Research in Open and Distributed Learning, 17(5). https://doi.org/10.19173/irrodl.v17i5.2350
31. Lin, L., Atkinson, R. K., Christopherson, R. M., Joseph, S. S., ve Harrison, C. J. (2013). Animated agents and learning: Does the type of verbal feedback they provide matter? Computers & Education, 67, 239-249. https://doi.org/10.1016/j.compedu.2013.04.017
32. Mayer, R. E. (1997). Multimedia learning: Are we asking the right questions? Educational Psychologist, 32(1), 1-19. https://doi.org/10.1207/s15326985ep3201_1
33. Mayer, R. E. (2002). Multimedia learning. İçinde Psychology of Learning and Motivation (C. 41, ss. 85-139). Academic Press. https://doi.org/10.1016/S0079-7421(02)80005-6
34. Mayer, R. E. (2014). Based Principles for Designing Multimedia Instruction. İçinde In their own words: What scholars and teachers want you to know about why and how to apply the science of learning in your academic setting. Society for the Teaching of Psychology.
35. Mayer, R. E., ve DaPra, C. S. (2012). An embodiment effect in computer-based learning with animated pedagogical agents. Journal of Experimental Psychology Applied, 18(3), 239-252. https://doi.org/10.1037/a0028616
36. Mayer, R. E., Sobko, K., ve Mautone, P. D. (2003). Social cues in multimedia learning: Role of speaker’s voice. Journal of Educational Psychology, 95(2), 419-425. https://doi.org/10.1037/0022-0663.95.2.419
37. Mayer, R. E., Stull, A., DeLeeuw, K., Almeroth, K., Bimber, B., Chun, D., Bulger, M., Campbell, J., Knight, A., ve Zhang, H. (2009). Clickers in college classrooms: Fostering learning with questioning methods in large lecture classes. Contemporary Educational Psychology, 34(1), 51-57. https://doi.org/10.1016/j.cedpsych.2008.04.002
38. Moreno, R., ve Mayer, R. (2007). Interactive Multimodal Learning Environments. Educational Psychology Review, 19(3), 309-326. https://doi.org/10.1007/s10648-007-9047-2
39. Moreno, R., Mayer, R. E., Spires, H. A., ve Lester, J. C. (2001). The Case for Social Agency in Computer-Based Teaching: Do Students Learn More Deeply When They Interact With Animated Pedagogical Agents? Cognition and Instruction, 19(2), 177-213. https://doi.org/10.1207/S1532690XCI1902_02
40. Mori, M., MacDorman, K. F., ve Kageki, N. (2012). The Uncanny Valley [From the Field]. IEEE Robotics & Automation Magazine, 19(2), 98-100. IEEE Robotics & Automation Magazine. https://doi.org/10.1109/MRA.2012.2192811
41. Olszanowski, M., Pochwatko, G., Kuklinski, K., Scibor-Rylski, M., Lewinski, P., ve Ohme, R. K. (2015). Warsaw set of emotional facial expression pictures: a validation study of facial display photographs. Frontiers in Psychology, 5. https://www.frontiersin.org/articles/10.3389/fpsyg.2014.01516
42. O’Toole, A. J., ve Abdi, H. (2001). Face Recognition Models. İçinde N. J. Smelser ve P. B. Baltes (Ed.), International Encyclopedia of the Social & Behavioral Sciences (ss. 5223-5226). Pergamon. https://doi.org/10.1016/B0-08-043076-7/00690-2
43. Paivio, A. (1990). Mental Representations: A Dual Coding Approach. Oxford University Press.
44. Patrick, F. (2023, Temmuz 27). Yüz algılama ve öznitelikler - Yüz - Azure AI services. https://learn.microsoft.com/tr-tr/azure/ai-services/computer-vision/concept-face-detection
45. Pekrun, R., ve Linnenbrink-Garcia, L. (2014). Introduction to Emotions in Education. İçinde International Handbook of Emotions in Education. Routledge.
46. Pickover, C. A. (2012). Phrenology 1796. İçinde The medical book: from witch doctors to robot surgeons, 250 milestones in the history of medicine. Union Square+ ORM.
47. Rasouli, A., Kotseruba, I., Kunic, T., ve Tsotsos, J. K. (2019). PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction. 6262-6271. https://openaccess.thecvf.com/content_ICCV_2019/html/Rasouli_PIE_A_Large-Scale_Dataset_and_Models_for_Pedestrian_Intention_Estimation_ICCV_2019_paper.html
48. Reeves, B., ve Nass, C. I. (1996). The media equation: How people treat computers, television, and new media like real people and places (ss. xiv, 305). Cambridge University Press.
49. Rosenberg-Kima, R. B., Plant, E. A., Doerr, C. E., ve Baylor, A. L. (2010). The Influence of Computer-based Model’s Race and Gender on Female Students’ Attitudes and Beliefs Towards Engineering. Journal of Engineering Education, 99(1), 35-44. https://doi.org/10.1002/j.2168-9830.2010.tb01040.x
50. Rowe, A. D., ve Fitness, J. (2018). Understanding the Role of Negative Emotions in Adult Learning and Achievement: A Social Functional Perspective. Behavioral Sciences, 8(2), Article 2. https://doi.org/10.3390/bs8020027
51. Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161-1178. https://doi.org/10.1037/h0077714
52. Scherer, K. R., ve Ekman, P. (1982). Handbook of methods in nonverbal behavior research. Cambridge University Press. https://cir.nii.ac.jp/crid/1130000796865038208
53. Schmidt, K. L., ve Cohn, J. F. (2001). Human facial expressions as adaptations: Evolutionary questions in facial expression research. American Journal of Physical Anthropology, 116(S33), 3-24. https://doi.org/10.1002/ajpa.20001
54. Sweller, J., Chandler, P., Tierney, P., ve Cooper, M. (1990). Cognitive load as a factor in the structuring of technical material. Journal of Experimental Psychology: General, 119(2), 176-192. https://doi.org/10.1037/0096-3445.119.2.176
55. Taxipulati, S., ve Lu, H.-D. (2021). The Influence of Feedback Content and Feedback Time on Multimedia Learning Achievement of College Students and Its Mechanism. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.706821
56. Thomas, R. A., West, R. E., ve Borup, J. (2017). An analysis of instructor social presence in online text and asynchronous video feedback comments. The Internet and Higher Education, 33, 61-73. https://doi.org/10.1016/j.iheduc.2017.01.003
57. Tonguç, G., ve Ozaydın Ozkara, B. (2020). Automatic recognition of student emotions from facial expressions during a lecture. Computers & Education, 148, 103797. https://doi.org/10.1016/j.compedu.2019.103797
58. Torres, D., Pulukuri, S., ve Abrams, B. (2022). Embedded Questions and Targeted Feedback Transform Passive Educational Videos into Effective Active Learning Tools. Journal of Chemical Education, 99(7), 2738-2742. https://doi.org/10.1021/acs.jchemed.2c00342
59. Tsai, M.-J., Hou, H.-T., Lai, M.-L., Liu, W.-Y., ve Yang, F.-Y. (2012). Visual attention for solving multiple-choice science problem: An eye-tracking analysis. Computers & Education, 58(1), 375-385. https://doi.org/10.1016/j.compedu.2011.07.012
60. Vonikakis, V., ve Winkler, S. (2020). Identity-Invariant Facial Landmark Frontalization For Facial Expression Analysis. 2020 IEEE International Conference on Image Processing (ICIP), 2281-2285. https://doi.org/10.1109/ICIP40778.2020.9190989
61. Vonikakis, V., ve Winkler, S. (2021). Efficient Facial Expression Analysis For Dimensional Affect Recognition Using Geometric Features (No. arXiv:2106.07817). arXiv. https://doi.org/10.48550/arXiv.2106.07817
62. Wang, C.-C., ve Yeh, W.-J. (2013). Avatars with Sex Appeal as Pedagogical Agents: Attractiveness, Trustworthiness, Expertise, and Gender Differences. Journal of Educational Computing Research, 48(4), 403-429. https://doi.org/10.2190/EC.48.4.a
63. Young, A. W. (2016). Facial Expression Recognition: Selected works of Andy Young. Psychology Press. https://doi.org/10.4324/9781315715933
64. Zel, S. (2020). Engaging Learners in Synchronous Online Training Using Facial Expression Analysis Technology [Ph.D.]. https://www.proquest.com/docview/2468167077/abstract/D075AEFF9455498FPQ/1