Measuring Cognitive Engagement: An Overview of Measurement Instruments and Techniques

Shan Li

Measuring Cognitive Engagement: An Overview of Measurement Instruments and Techniques

Abstract

This paper adopted an analytical perspective to review cognitive engagement measures. This paper provided a comprehensive understanding of the instruments/techniques used to measure cognitive engagement, which could assist researchers or practitioners in improving their measurement methodologies. In particular, we conducted a systematic literature search, based on which the current practice in measuring cognitive engagement was synthesized. We organized and aggregated the information of cognitive engagement measures by their types, including self-report scales, observations, interviews, teacher ratings, experience sampling, eye-tracking, physiological sensors, trace analysis, and content analysis. We provided a critical analysis of the strength and weaknesses of each measurement method. Recommendations for measuring cognitive engagement were also provided to guide future empirical work in a meaningful direction.

Keywords

References

Antonietti, A., Colombo, B., & Nuzzo, C. Di. (2015). Metacognition in self-regulated multimedia learning: Integrating behavioural, psychophysiological and introspective measures. Learning, Media and Technology, 40(2), 187–209. http://dx.doi.org/10.1080/17439884.2014.933112
Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of School Psychology, 44(5), 427–445. https://doi.org/10.1016/j.jsp.2006.04.002
Azevedo, R. (2015). Defining and measuring engagement and learning in science: Conceptual, theoretical, methodological, and analytical issues. Educational Psychologist, 50(1), 84–94. https://doi.org/10.1080/00461520.2015.1004069
Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V. T., … Craven, P. L. (2007). EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, Space, and Environmental Medicine, 78(5), B231–B244. https://doi.org/10.1016/j.biopsycho.2011.03.003
Bernacki, M. L., Byrnes, J. P., & Cromley, J. G. (2012). The effects of achievement goals and self-regulated learning behaviors on reading comprehension in technology-enhanced learning environments. Contemporary Educational Psychology, 37(2), 148–161. https://doi.org/10.1016/j.cedpsych.2011.12.001
Betts, J. (2012). Issues and methods in the measurement of student engagement: Advancing the construct through statistical modeling. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement (pp. 783–803). Springer Science + Business Media. https://doi.org/10.1007/978-1-4614-2018-7_38
Charland, P., Léger, P.-M., Sénécal, S., Courtemanche, F., Mercier, J., Skelling, Y., & Labonté-Lemoyne, E. (2015). Assessing the multiple dimensions of engagement to characterize learning: A neurophysiological perspective. Journal of Visualized Experiments, (101), 1–8. https://doi.org/10.3791/52627
Chen, C.-Y., & Pedersen, S. (2012). Learners’ internal management of cognitive processing in online learning. Innovations in Education and Teaching International, 49(4), 363–373. http://dx.doi.org/10.1080/14703297.2012.728873

Cleary, T. J., & Zimmerman, B. J. (2012). A cyclical self-regulatory account of student engagement- theoretical Foundations and Applications. In S. L. Christenson, A. Wylie, & C. Reschly (Eds.), Handbook of Research on Student Engagement (pp. 237–257). Springer Science+Business Media. https://doi.org/10.1007/978-1-4614-2018-7
D’Mello, S., Dieterle, E., & Duckworth, A. (2017). Advanced, analytic, automated (AAA) measurement of engagement during learning. Educational Psychologist, 52(2), 104–123. https://doi.org/10.1080/00461520.2017.1281747
Dent, A. L., & Koenka, A. C. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: A meta-analysis. Educational Psychology Review, 28(3), 425–474. https://doi.org/10.1007/s10648-015-9320-8
Finlay, K. a. (2006). Quantifying school engagement: Research report. Denver, CO.
Finn, J. D., & Zimmer, K. S. (2012). Student engagement: What is it? Why soes it matter? In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement (pp. 97–131). Boston, MA: Springer Science+Business Media. https://doi.org/10.1007/978-1-4614-2018-7_5
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109. https://doi.org/10.3102/00346543074001059
Fredricks, J. A., Blumenfeld, P., Friedel, J., & Paris, A. (2005). School engagement. In K. A. Moore & L. Laura (Eds.), Conceptualizing and measuring indicators of positive development: What do children need to flourish (pp. 305–321). Kluwer academic/plenum press. https://doi.org/10.1207/s15327752jpa8502
Fredricks, J. A., & Mccolskey, W. (2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement (pp. 763–782). Boston, MA: Springer Science + Business Media. https://doi.org/10.1007/978-1-4614-2018-7
Fredricks, J., McColskey, W., Meli, J., Mordica, J., Montrosse, B., & Mooney, K. (2011). Measuring student engagement in upper elementary through high school: A description of 21 instruments. Issues & Answers. REL 2011-No. 098. Regional Educational Laboratory Southeast.
Furlong, M. J., & Christenson, S. L. (2008). Engaging students at school and with learning: A relevant construct for all students. Psychology in the Schools. https://doi.org/10.1002/pits.20302
Greene, B. A. (2015). Measuring cognitive engagement with self-report scales: Reflections from over 20 Years of research. Educational Psychologist, 50(1), 14–30. https://doi.org/10.1080/00461520.2014.989230
Greene, B. A., Miller, R. B., Crowson, H. M., Duke, B. L., & Akey, K. L. (2004). Predicting high school students’ cognitive engagement and achievement: Contributions of classroom perceptions and motivation. Contemporary Educational Psychology, 29(4), 462–482. https://doi.org/10.1016/j.cedpsych.2004.01.006
Helme, S., & Clarke, D. (2001). Identifying cognitive engagement in the mathematics classroom. Mathematics Education Research Journal, 13(2), 133–153. https://doi.org/10.1007/BF03217103
Ireland, M. E., & Henderson, M. D. (2014). Language style matching, engagement, and impasse in negotiations. Negotiation and Conflict Management Research, 7(1), 1–16.
Järvelä, S., Veermans, M., & Leinonen, P. (2008). Investigating student engagement in computer-supported inquiry: A process-oriented analysis. Social Psychology of Education, 11(3), 299–322. https://doi.org/10.1007/s11218-007-9047-6
Kong, Q. P., Wong, N. Y., & Lam, C. C. (2003). Student engagement in mathematics: Development of instrument and validation of construct. Mathematics Education Research Journal, 15(1), 4–21. https://doi.org/10.1007/BF03217366
Kruger, J., Hefer, E., & Matthew, G. (2014). Attention distribution and cognitive load in a subtitled academic lecture: L1 vs. L2. Journal of Eye Movement Research, 7(5), 1–15. https://doi.org/10.16910/jemr.7.5.4
Lee, O., & Anderson, C. W. (1993). Task engagement and conceptual change in middle school science classrooms. American Educational Research Journal, 30(3), 585–610.
Lee, W., & Reeve, J. (2012). Teachers’ estimates of their students’ motivation and engagement: Being in synch with students. Educational Psychology, 32(6), 727–747. https://doi.org/10.1080/01443410.2012.732385
Li, S., Lajoie. S.P., Zheng, J., Wu, H., & Cheng, H. (2021). Automated detection of cognitive engagement to inform the art of staying engaged in problem-solving. Computers and Education. 163, 104114.
Li, S., Zheng, J., Poitras, E., & Lajoie, S. (2018). The allocation of time matters to students’ performance in clinical reasoning. In R. Nkambou, R. Azevedo, & J. Vassileva (Eds.), Lecture notes in computer sciences (pp. 110–119). Springer International Publishing.
Linnenbrink, E. A. (2005). The dilemma of performance-approach goals: The use of multiple goal contexts to promote students’ motivation and learning. Journal of Educational Psychology, 97(2), 197–213. https://doi.org/10.1037/0022-0663.97.2.197
Meece, J. L., Blumenfeld, P. C., & Hoyle, R. H. (1988). Students’ goal orientations and cognitive engagement in classroom activities. Journal of Educational Psychology, 80(4), 514–523. https://doi.org/10.1037/0022-0663.80.4.514
Miller, B. W. (2015). Using reading times and eye-movements to measure cognitive engagement. Educational Psychologist, 50(1), 31–42. https://doi.org/10.1080/00461520.2015.1004068
Patrick, H., Ryan, A. M., & Kaplan, A. (2007). Early adolescents’ perceptions of the classroom social environment, motivational beliefs, and engagement. Journal of Educational Psychology, 99(1), 83–98.
Pennebaker, J. W., Boyd, R. L., Jordan, K., & Blackburn, K. (2015). The development and psychometric properties of LIWC2015. Austin, TX.
Pintrich, P. R., & de Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82(1), 33–40. https://doi.org/10.1037/0022-0663.82.1.33
Richardson, J. C., & Newby, T. (2006). The role of students’ cognitive engagement in online learning. American Journal of Distance Education, 20(1), 23–37. https://doi.org/10.1207/s15389286ajde2001
Rotgans, J. I., & Schmidt, H. G. (2011). Cognitive engagement in the problem-based learning classroom. Advances in Health Sciences Education, 16(4), 465–479. https://doi.org/10.1007/s10459-011-9272-9
Salmela-Aro, K., Moeller, J., Schneider, B., Spicer, J., & Lavonen, J. (2016). Integrating the light and dark sides of student engagement using person-oriented and situation-specific approaches. Learning and Instruction, 43, 61–70. https://doi.org/10.1016/j.learninstruc.2016.01.001
Samuelsen, K. M. (2012). Part V commentary: Possible new directions in the measurement of student engagement. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement (pp. 805–811). Boston, MA: Springer Science + Business Media. https://doi.org/10.1007/978-1-4614-2018-7_39
Schaufeli, Martínez, I. M., Pinto, A. M., Salanova, M., & Bakker, A. B. (2002). Burnout and engagement in university students. Journal of Cross-Cultural Psychology, 33(5), 464–481. https://doi.org/10.1177/0022022102033005003
Schuurink, E. L., Houtkamp, J., & Toet, A. (2008). Engagement and EMG in serious gaming: Experimenting with sound and dynamics in the levee patroller training game. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5294 LNCS, 139–149. https://doi.org/10.1007/978-3-540-88322-7-14
Shernoff, D. J., Kelly, S., Tonks, S. M., Anderson, B., Cavanagh, R. F., Sinha, S., & Abdi, B. (2016). Student engagement as a function of environmental complexity in high school classrooms. Learning and Instruction, 43, 52–60. https://doi.org/10.1016/j.learninstruc.2015.12.003
Sinatra, G. M., Heddy, B. C., & Lombardi, D. (2015). The challenges of defining and Mmeasuring student engagement in science. Educational Psychologist, 50(1). https://doi.org/10.1080/00461520.2014.1002924
Stevens, R., Galloway, T., & Berka, C. (2007). EEG-related changes in cognitive workload, engagement and distraction as students acquire problem solving skills. In 11th International Conference on User Modeling (UM 2007) (pp. 187–196). https://doi.org/10.1007/978-3-540-73078-1_22
Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24–54. https://doi.org/10.1177/0261927X09351676
Turner, J. C., & Meyer, D. K. (2000). Studying and understanding the instructional contexts of classrooms: Using our past to forge our future. Educational Psychologist, 35(2), 69–85. https://doi.org/10.1207/S15326985EP3502
van Gog, T., & Jarodzka, H. (2013). Eye tracking as a tool to study and enhance cognitive and metacognitive processes in computer-based learning environments. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning Technologies (Vol. 28, pp. 143–156). Springer Science+Business Media. https://doi.org/10.1007/978-1-4419-5546-3
Veiga, F. H., Reeve, J., Wentzel, K., & Robu, V. (2014). Assessing students’ engagement: A review of instruments with psychometric Qualities. Students’ Engagement in School: International Perspectives of Psychology and Education, 38–57.
Wigfield, A., Guthrie, J. T., Perencevich, K. C., Taboada, A., Klauda, S. L., McRae, A., & Barbosa, P. (2008). Role of reading engagement in mediating the effects of reading comprehension instruction on reading outcomes. Psychology in the Schools, 45(5), 432–445. https://doi.org/10.1002/pits
Wolters, C. A. (2004). Advancing achievement goal theory: Using goal structures and goal orientations to predict students’ motivation, cognition, and achievement. Journal of Educational Psychology, 96(2), 236–250. http://10.0.4.13/0022-0663.96.2.236
Xie, K., Heddy, B. C., & Greene, B. A. (2018). Affordances of using mobile technology to support experience-sampling method in examining college students’ engagement. Computers & Education. https://doi.org/10.1016/J.COMPEDU.2018.09.020
Zhu, E. (2006). Interaction and cognitive engagement : An analysis of four asynchronous online discussions. Instructional Science, 34, 451–480. https://doi.org/10.1007/s11251-006-0004-0
Zimmerman, B. J., & Martinez-Pons, M. (1986). Development of a structured interview for assessing student use of self-regulated learning strategies. American Educational Research Journal, 23(4), 614–628. https://doi.org/10.2307/1163093
Zirkel, S., Garcia, J. A., & Murphy, M. C. (2015). Experience-sampling research methods and their potential for education research. Educational Researcher, 44(1), 7–16. https://doi.org/10.3102/0013189X14566879

Details

Primary Language

English

Subjects

Other Fields of Education

Journal Section

Research Article

Authors

Shan Li This is me
0000-0001-6001-1586
Canada

Publication Date

July 25, 2021

Submission Date

December 1, 2020

Acceptance Date

May 16, 2021

Published in Issue

Year 2021 Volume: 8 Number: 3

IZ

https://izlik.org/JA97WT89UC

Cite

RIS / Bibtex

APA

Li, S. (2021). Measuring Cognitive Engagement: An Overview of Measurement Instruments and Techniques. International Journal of Psychology and Educational Studies, 8(3), 63-76. https://izlik.org/JA97WT89UC