Comparison of Methods of Affect Transition Analysis: An Example of SimInClass Dataset

Abstract

Studies of emotional-cognitive sequences are the growing body of research area in educational context. These studies focus on how emotions change during the learning-teaching process due to their dynamic nature. In affect transition studies, the change of emotion, depending on the event and time, is usually analyzed by using (a) lag sequential analysis (LSA), (b) L metric, (c) L* metric, and (d) Yule's Q metric. Yet, various methodological criticisms exist in the literature while utilizing these sequential analysis methods. In this study, it is aimed to comparatively examine lag analysis, L metric, L* metric, and Yule’s Q in terms of proportion of invalid values, maximum transition metrics, minimum transition metrics, and analysis results. For this reason, the emotional states of the fifteen prospective teachers were collected and their emotions were labeled every 0.5 seconds with EEG (Electroencephalogram), GSR (Galvanic Skin Response), and Microsoft Kinect in a teacher training simulator (SimInClass). The dataset contained 17570 emotions, and the data were analyzed by utilizing lag analysis, L, L* and Yule's Q. The results showed that LSA had yielded the most proportion of invalid results. In addition, it was observed that the number of invalid values increased as the segment length became shorter in all analysis methods. When the maximum and minimum transition metric values were examined, it was found that as the sequence length increased in L and L* analyses, the value of the metrics approached 1, which is the largest value they can reach. However, it was noted that the lag analysis maximum-minimum transition metrics fluctuate independently from the sequence length. It was concluded that there were differences in the analysis results produced by the four sequential analysis methods with the same functions. It was thought that this situation might be due to the different invalid results produced by the analyses. When the results were compared with the studies in the literature, it was thought that it would be beneficial to pay attention to the nature of the data (emotional or behavioral), the data type (singe modality or multimodal modality), the amount of data (short sequences or long sequences), the environment in which the dataset was created (computer-based or not), and the sampling rate (automated data collection tool or observation) when choosing sequential analysis methods.

Keywords

• affect
• affect transition
• sequential analysis

References

1. Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis. Cambridge University Press. https://doi.org/10.1017/CBO9780511527685
2. Baker, R. S., D'Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223-241. https://doi.org/10.1016/j.ijhcs.2009.12.003
3. Baker, R. S., Rodrigo, M. M. T., & Xolocotzin, U. E. (2007). The dynamics of affective transitions in simulation problem-solving environments. In A. Paiva, R. Prada & W. Picard (Eds.), International conference on affective computing and intelligent interaction (pp. 666-677). Springer. https://doi.org/10.1007/978-3-540-74889-2_58
4. Bayazıt, T. (2018). Event Related Potentials (ERP). Journal of Medical Clinics, 1(1), 59-65. https://dergipark.org.tr/tr/pub/atk/issue/38771/451155
5. Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of statistics, 29(4),1165-1188. https://doi.org/10.1214/aos/1013699998
6. Bosch, N., & D’Mello, S. (2017). The affective experience of novice computer programmers. International Journal of Artificial Intelligence in Education, 27(1), 181-206. https://doi.org/10.1007/s40593-015-0069-5
7. Bosch, N., & Paquette, L. (2021). What’s next? Sequence length and impossible loops in state transition measurement. Journal of Educational Data Mining, 13(1), 1-23. https://eric.ed.gov/?id=EJ1320638
8. Botelho, A. F., Baker, R., Ocumpaugh, J., & Heffernan, N. (2018). Studying affect dynamics and chronometry using sensor-free detectors. In E. Boyer & M. Yudelson (Eds.), Proceedings of the 11th international conference on educational data mining (pp. 157–166). EDM. https://files.eric.ed.gov/fulltext/ED593106.pdf

9. Cimtay, Y., Ekmekcioglu, E., & Caglar-Ozhan, S. (2020). Cross-subject multimodal emotion recognition based on hybrid fusion. IEEE Access, 8, 168865-168878. https://doi.org/10.1109/ACCESS.2020.3023871
10. D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145-157. https://doi.org/10.1016/j.learninstruc.2011.10.001
11. D’Mello, S., Taylor, R., & Graesser, A. (2007). Monitoring affective trajectories during complex learning. In McNamara, D., Trafton, J. (Eds.), Proceedings of 29th annual cognitive science society (pp. 203–208). Cognitive Science Society. https://doi.org/10.1007/978-1-4419-1428-6_849
12. Ekman (2021, September 7). What is surprise? Paul Ekman. https://www.paulekman.com/universal-emotions/what-is-surprise/
13. Frenzel, A. C., Goetz, T., Stephens, E. J., & Jacob, B. (2009). Antecedents and effects of teachers’ emotional experiences: An integrated perspective and empirical test. In P. A. Schutz & M. Zembylas (Eds.), Advances in teacher emotion research: The impact on teachers’ lives (pp. 129-152). Springer. https://doi.org/10.1007/978-1-4419-0564-2_7
14. Han, J.-H., Shubeck, K., Shi, G.-H., Hu, X.-E., Yang, L., Wang, L.-J., Zhao, W., Jiang, Q., & Biswas, G. (2021). Teachable agent improves affect regulation: Evidence from Betty’s brain. Educational Technology & Society, 24(3), 194–209. https://www.jstor.org/stable/27032865
15. Juslin, P. N., & Sloboda, J. A. (2013). Music and emotion. In D. Deutsch (Ed.), The psychology of music (pp. 583-645). Academic Press. https://doi.org/10.1016/B978-0-12-381460-9.00015-8
16. Karumbaiah, S., Baker, R. S., & Ocumpaugh, J. (2019). The case of self-transitions in affective dynamics. In S. Isotani, E. Millán, A. Ogan, P. Hastings, B. McLaren, & R. Luckin (Eds.), International conference on artificial intelligence in education (pp. 172-181). Springer. https://doi.org/10.1007/978-3-030-23204-7_15
17. Karumbaiah, S., Baker, R. B., Ocumpaugh, J., & Andres, A. (2021). A re-analysis and synthesis of data on affect dynamics in learning. IEEE Transactions on Affective Computing. https://doi.org/10.1109/TAFFC.2021.3086118
18. Kleinginna, P. R., & Kleinginna, A. M. (1981). A categorized list of emotion definitions, with suggestions for a consensual definition. Motivation and Emotion, 5(4), 345-379. https://doi.org/10.1007/BF00992553
19. Lajoie, S. P., Zheng, J., Li, S., Jarrell, A., & Gube, M. (2021). Examining the interplay of affect and self regulation in the context of clinical reasoning. Learning and Instruction, 72, 101219. https://doi.org/10.1016/j.learninstruc.2019.101219
20. Liu, Z., Zhang, N., Liu, S., & Liu, S. (2021). Development trajectory of student cognitive behaviors in a SPOC forum: An integrated approach combining epistemic network analysis and lag sequential analysis. In L. Lee, L. Wang, Y. Kato & S. Sato (Eds.), 2021 International symposium on educational technology (ISET) (pp. 26-30). IEEE. https://doi.org/10.1109/ISET52350.2021.00016.
21. Matayoshi, J., & Karumbaiah, S. (2020). Adjusting the L statistic when self-transitions are excluded in affect dynamics. Journal of Educational Data Mining, 12(4), 1-23. https://eric.ed.gov/?id=EJ1298368
22. Matayoshi, J., & Karumbaiah, S. (2021). Using marginal models to adjust for statistical bias in the analysis of state transitions. In M. Scheffel, N. Dowell, S. Joksimovic & G. Siemens (Eds.), LAK21: 11th International learning analytics and knowledge conference (pp. 449-455). Association for Computing Machinery. https://doi.org/10.1145/3448139.3448182
23. Rebolledo-Mendez, G., Huerta-Pacheco, N. S., Baker, R. S., & du Boulay, B. (2022). Meta-affective behaviour within an intelligent tutoring system for mathematics. International Journal of Artificial Intelligence in Education, 32(1), 174-195. https://doi.org/10.1007/s40593-021-00247-1
24. Pohl, M., Wallner, G., & Kriglstein, S. (2016). Using lag-sequential analysis for understanding interaction sequences in visualizations. International Journal of Human-Computer Studies, 96, 54-66. https://doi.org/10.1016/j.ijhcs.2016.07.006
25. Scherer, K. R. (1993). Studying the emotion-antecedent appraisal process: An expert system approach. Cognition and Emotion, 7(3), 325–355. https://doi.org/10.1080/02699939308409192
26. Sebe, N., Cohen, I., & Huang, T. S. (2005). Multimodal emotion recognition. In C. Chen & P. Wang (Eds.), Handbook of pattern recognition and computer vision (pp. 387-409). World Scientific. https://doi.org/10.1142/1802
27. Sun, J. C. Y., Kuo, C. Y., Hou, H. T., & Lin, Y. Y. (2017). Exploring learners' sequential behavioral patterns, flow experience, and learning performance in an anti-phishing educational game. Journal of Educational Technology & Society, 20(1), 10-20. https://www.proquest.com/scholarly-journals/exploring-learners-sequential-behavioral-patterns/docview/2147743221/se-2
28. Sun, Z., Lin, C. H., Lv, K., & Song, J. (2021). Knowledge-construction behaviors in a mobile learning environment: A lag-sequential analysis of group differences. Educational Technology Research and Development, 69(2), 533-551. https://doi.org/10.1007/s11423-021-09938-x
29. Wu, S. Y., & Hou, H. T. (2015). How cognitive styles affect the learning behaviors of online problem-solving based discussion activity: A lag sequential analysis. Journal of Educational Computing Research, 52(2), 277-298. https://eric.ed.gov/?id=EJ1076314
30. Yang, X., Song, S., Zhao, X., & Yu, S. (2018). Understanding user behavioral patterns in open knowledge communities. Interactive Learning Environments, 26(2), 245-255. https://doi.org/10.1080/10494820.2017.1303518
31. Yang, X., Li, J., & Xing, B. (2018). Behavioral patterns of knowledge construction in online cooperative translation activities. The Internet and Higher Education, 36, 13-21. https://doi.org/10.1016/j.iheduc.2017.08.003
32. Yule, G. U. (1900). On the association of attributes in statistics: With illustrations from the material of the childhood society. Philosophical Transactions of the Royal Society of London. 66(194), 252–261. https://doi.org/10.1098/rspl.1899.0067