Measuring Cognitive Engagement: An Overview of Measurement Instruments and Techniques

Year 2021, Volume: 8 Issue: 3, 63 - 76, 25.07.2021

Shan Li

Abstract

This paper adopted an analytical perspective to review cognitive engagement measures. This paper provided a comprehensive understanding of the instruments/techniques used to measure cognitive engagement, which could assist researchers or practitioners in improving their measurement methodologies. In particular, we conducted a systematic literature search, based on which the current practice in measuring cognitive engagement was synthesized. We organized and aggregated the information of cognitive engagement measures by their types, including self-report scales, observations, interviews, teacher ratings, experience sampling, eye-tracking, physiological sensors, trace analysis, and content analysis. We provided a critical analysis of the strength and weaknesses of each measurement method. Recommendations for measuring cognitive engagement were also provided to guide future empirical work in a meaningful direction.

Keywords

Cognitive engagement, measurement instruments and techniques, systematic literature search, research synthesis, multiple methods

References

• Antonietti, A., Colombo, B., & Nuzzo, C. Di. (2015). Metacognition in self-regulated multimedia learning: Integrating behavioural, psychophysiological and introspective measures. Learning, Media and Technology, 40(2), 187–209. http://dx.doi.org/10.1080/17439884.2014.933112
• Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of School Psychology, 44(5), 427–445. https://doi.org/10.1016/j.jsp.2006.04.002
• Azevedo, R. (2015). Defining and measuring engagement and learning in science: Conceptual, theoretical, methodological, and analytical issues. Educational Psychologist, 50(1), 84–94. https://doi.org/10.1080/00461520.2015.1004069
• Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V. T., … Craven, P. L. (2007). EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, Space, and Environmental Medicine, 78(5), B231–B244. https://doi.org/10.1016/j.biopsycho.2011.03.003
• Bernacki, M. L., Byrnes, J. P., & Cromley, J. G. (2012). The effects of achievement goals and self-regulated learning behaviors on reading comprehension in technology-enhanced learning environments. Contemporary Educational Psychology, 37(2), 148–161. https://doi.org/10.1016/j.cedpsych.2011.12.001
• Betts, J. (2012). Issues and methods in the measurement of student engagement: Advancing the construct through statistical modeling. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement (pp. 783–803). Springer Science + Business Media. https://doi.org/10.1007/978-1-4614-2018-7_38
• Charland, P., Léger, P.-M., Sénécal, S., Courtemanche, F., Mercier, J., Skelling, Y., & Labonté-Lemoyne, E. (2015). Assessing the multiple dimensions of engagement to characterize learning: A neurophysiological perspective. Journal of Visualized Experiments, (101), 1–8. https://doi.org/10.3791/52627
• Chen, C.-Y., & Pedersen, S. (2012). Learners’ internal management of cognitive processing in online learning. Innovations in Education and Teaching International, 49(4), 363–373. http://dx.doi.org/10.1080/14703297.2012.728873
• Cleary, T. J., & Zimmerman, B. J. (2012). A cyclical self-regulatory account of student engagement- theoretical Foundations and Applications. In S. L. Christenson, A. Wylie, & C. Reschly (Eds.), Handbook of Research on Student Engagement (pp. 237–257). Springer Science+Business Media. https://doi.org/10.1007/978-1-4614-2018-7
• D’Mello, S., Dieterle, E., & Duckworth, A. (2017). Advanced, analytic, automated (AAA) measurement of engagement during learning. Educational Psychologist, 52(2), 104–123. https://doi.org/10.1080/00461520.2017.1281747
• Dent, A. L., & Koenka, A. C. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: A meta-analysis. Educational Psychology Review, 28(3), 425–474. https://doi.org/10.1007/s10648-015-9320-8
• Finlay, K. a. (2006). Quantifying school engagement: Research report. Denver, CO.
• Finn, J. D., & Zimmer, K. S. (2012). Student engagement: What is it? Why soes it matter? In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement (pp. 97–131). Boston, MA: Springer Science+Business Media. https://doi.org/10.1007/978-1-4614-2018-7_5
• Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109. https://doi.org/10.3102/00346543074001059
• Fredricks, J. A., Blumenfeld, P., Friedel, J., & Paris, A. (2005). School engagement. In K. A. Moore & L. Laura (Eds.), Conceptualizing and measuring indicators of positive development: What do children need to flourish (pp. 305–321). Kluwer academic/plenum press. https://doi.org/10.1207/s15327752jpa8502
• Fredricks, J. A., & Mccolskey, W. (2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement (pp. 763–782). Boston, MA: Springer Science + Business Media. https://doi.org/10.1007/978-1-4614-2018-7
• Fredricks, J., McColskey, W., Meli, J., Mordica, J., Montrosse, B., & Mooney, K. (2011). Measuring student engagement in upper elementary through high school: A description of 21 instruments. Issues & Answers. REL 2011-No. 098. Regional Educational Laboratory Southeast.
• Furlong, M. J., & Christenson, S. L. (2008). Engaging students at school and with learning: A relevant construct for all students. Psychology in the Schools. https://doi.org/10.1002/pits.20302
• Greene, B. A. (2015). Measuring cognitive engagement with self-report scales: Reflections from over 20 Years of research. Educational Psychologist, 50(1), 14–30. https://doi.org/10.1080/00461520.2014.989230
• Greene, B. A., Miller, R. B., Crowson, H. M., Duke, B. L., & Akey, K. L. (2004). Predicting high school students’ cognitive engagement and achievement: Contributions of classroom perceptions and motivation. Contemporary Educational Psychology, 29(4), 462–482. https://doi.org/10.1016/j.cedpsych.2004.01.006
• Helme, S., & Clarke, D. (2001). Identifying cognitive engagement in the mathematics classroom. Mathematics Education Research Journal, 13(2), 133–153. https://doi.org/10.1007/BF03217103
• Ireland, M. E., & Henderson, M. D. (2014). Language style matching, engagement, and impasse in negotiations. Negotiation and Conflict Management Research, 7(1), 1–16.
• Järvelä, S., Veermans, M., & Leinonen, P. (2008). Investigating student engagement in computer-supported inquiry: A process-oriented analysis. Social Psychology of Education, 11(3), 299–322. https://doi.org/10.1007/s11218-007-9047-6
• Kong, Q. P., Wong, N. Y., & Lam, C. C. (2003). Student engagement in mathematics: Development of instrument and validation of construct. Mathematics Education Research Journal, 15(1), 4–21. https://doi.org/10.1007/BF03217366
• Kruger, J., Hefer, E., & Matthew, G. (2014). Attention distribution and cognitive load in a subtitled academic lecture: L1 vs. L2. Journal of Eye Movement Research, 7(5), 1–15. https://doi.org/10.16910/jemr.7.5.4
• Lee, O., & Anderson, C. W. (1993). Task engagement and conceptual change in middle school science classrooms. American Educational Research Journal, 30(3), 585–610.
• Lee, W., & Reeve, J. (2012). Teachers’ estimates of their students’ motivation and engagement: Being in synch with students. Educational Psychology, 32(6), 727–747. https://doi.org/10.1080/01443410.2012.732385
• Li, S., Lajoie. S.P., Zheng, J., Wu, H., & Cheng, H. (2021). Automated detection of cognitive engagement to inform the art of staying engaged in problem-solving. Computers and Education. 163, 104114.
• Li, S., Zheng, J., Poitras, E., & Lajoie, S. (2018). The allocation of time matters to students’ performance in clinical reasoning. In R. Nkambou, R. Azevedo, & J. Vassileva (Eds.), Lecture notes in computer sciences (pp. 110–119). Springer International Publishing.
• Linnenbrink, E. A. (2005). The dilemma of performance-approach goals: The use of multiple goal contexts to promote students’ motivation and learning. Journal of Educational Psychology, 97(2), 197–213. https://doi.org/10.1037/0022-0663.97.2.197
• Meece, J. L., Blumenfeld, P. C., & Hoyle, R. H. (1988). Students’ goal orientations and cognitive engagement in classroom activities. Journal of Educational Psychology, 80(4), 514–523. https://doi.org/10.1037/0022-0663.80.4.514
• Miller, B. W. (2015). Using reading times and eye-movements to measure cognitive engagement. Educational Psychologist, 50(1), 31–42. https://doi.org/10.1080/00461520.2015.1004068
• Patrick, H., Ryan, A. M., & Kaplan, A. (2007). Early adolescents’ perceptions of the classroom social environment, motivational beliefs, and engagement. Journal of Educational Psychology, 99(1), 83–98.
• Pennebaker, J. W., Boyd, R. L., Jordan, K., & Blackburn, K. (2015). The development and psychometric properties of LIWC2015. Austin, TX.
• Pintrich, P. R., & de Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82(1), 33–40. https://doi.org/10.1037/0022-0663.82.1.33
• Richardson, J. C., & Newby, T. (2006). The role of students’ cognitive engagement in online learning. American Journal of Distance Education, 20(1), 23–37. https://doi.org/10.1207/s15389286ajde2001
• Rotgans, J. I., & Schmidt, H. G. (2011). Cognitive engagement in the problem-based learning classroom. Advances in Health Sciences Education, 16(4), 465–479. https://doi.org/10.1007/s10459-011-9272-9
• Salmela-Aro, K., Moeller, J., Schneider, B., Spicer, J., & Lavonen, J. (2016). Integrating the light and dark sides of student engagement using person-oriented and situation-specific approaches. Learning and Instruction, 43, 61–70. https://doi.org/10.1016/j.learninstruc.2016.01.001
• Samuelsen, K. M. (2012). Part V commentary: Possible new directions in the measurement of student engagement. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement (pp. 805–811). Boston, MA: Springer Science + Business Media. https://doi.org/10.1007/978-1-4614-2018-7_39
• Schaufeli, Martínez, I. M., Pinto, A. M., Salanova, M., & Bakker, A. B. (2002). Burnout and engagement in university students. Journal of Cross-Cultural Psychology, 33(5), 464–481. https://doi.org/10.1177/0022022102033005003
• Schuurink, E. L., Houtkamp, J., & Toet, A. (2008). Engagement and EMG in serious gaming: Experimenting with sound and dynamics in the levee patroller training game. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5294 LNCS, 139–149. https://doi.org/10.1007/978-3-540-88322-7-14
• Shernoff, D. J., Kelly, S., Tonks, S. M., Anderson, B., Cavanagh, R. F., Sinha, S., & Abdi, B. (2016). Student engagement as a function of environmental complexity in high school classrooms. Learning and Instruction, 43, 52–60. https://doi.org/10.1016/j.learninstruc.2015.12.003
• Sinatra, G. M., Heddy, B. C., & Lombardi, D. (2015). The challenges of defining and Mmeasuring student engagement in science. Educational Psychologist, 50(1). https://doi.org/10.1080/00461520.2014.1002924
• Stevens, R., Galloway, T., & Berka, C. (2007). EEG-related changes in cognitive workload, engagement and distraction as students acquire problem solving skills. In 11th International Conference on User Modeling (UM 2007) (pp. 187–196). https://doi.org/10.1007/978-3-540-73078-1_22
• Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24–54. https://doi.org/10.1177/0261927X09351676
• Turner, J. C., & Meyer, D. K. (2000). Studying and understanding the instructional contexts of classrooms: Using our past to forge our future. Educational Psychologist, 35(2), 69–85. https://doi.org/10.1207/S15326985EP3502
• van Gog, T., & Jarodzka, H. (2013). Eye tracking as a tool to study and enhance cognitive and metacognitive processes in computer-based learning environments. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning Technologies (Vol. 28, pp. 143–156). Springer Science+Business Media. https://doi.org/10.1007/978-1-4419-5546-3
• Veiga, F. H., Reeve, J., Wentzel, K., & Robu, V. (2014). Assessing students’ engagement: A review of instruments with psychometric Qualities. Students’ Engagement in School: International Perspectives of Psychology and Education, 38–57.
• Wigfield, A., Guthrie, J. T., Perencevich, K. C., Taboada, A., Klauda, S. L., McRae, A., & Barbosa, P. (2008). Role of reading engagement in mediating the effects of reading comprehension instruction on reading outcomes. Psychology in the Schools, 45(5), 432–445. https://doi.org/10.1002/pits
• Wolters, C. A. (2004). Advancing achievement goal theory: Using goal structures and goal orientations to predict students’ motivation, cognition, and achievement. Journal of Educational Psychology, 96(2), 236–250. http://10.0.4.13/0022-0663.96.2.236
• Xie, K., Heddy, B. C., & Greene, B. A. (2018). Affordances of using mobile technology to support experience-sampling method in examining college students’ engagement. Computers & Education. https://doi.org/10.1016/J.COMPEDU.2018.09.020
• Zhu, E. (2006). Interaction and cognitive engagement : An analysis of four asynchronous online discussions. Instructional Science, 34, 451–480. https://doi.org/10.1007/s11251-006-0004-0
• Zimmerman, B. J., & Martinez-Pons, M. (1986). Development of a structured interview for assessing student use of self-regulated learning strategies. American Educational Research Journal, 23(4), 614–628. https://doi.org/10.2307/1163093
• Zirkel, S., Garcia, J. A., & Murphy, M. C. (2015). Experience-sampling research methods and their potential for education research. Educational Researcher, 44(1), 7–16. https://doi.org/10.3102/0013189X14566879