Self-regulated learning support in technology enhanced learning environments: A reliability analysis of the SRL-S rubric

Abstract

Advanced learning technologies have become a focal point in recent educational research, holding the promise of enhancing students' self-regulated learning (SRL) by facilitating various processes of planning, monitoring, performing, and reflecting upon learning experiences. However, concerns have arisen regarding the efficacy and design of technologies, the spectrum of possibilities for SRL support, and too ambiguous claims associated with these technologies. To address these uncertainties and to provide a platform for generating the more empirical evidence, Self-Regulated Learning Support (SRL-S) rubric was developed to facilitate the assessment of SRL support in technology-enhanced learning environments. It is grounded in established educational theory and proven empirical research results. This article presents a study that extends the application of the rubric to establish its reliability and validity, filling a gap in prior research. First, content, criterion-related, and construct validation were performed through international and interdisciplinary experts’ reviews. Subsequently, inter-rater and intra-rater reliability were assessed using Intraclass Correlation Coefficients and Cohens Kappa tests. The outcomes of these analysis demonstrated that the SRL-S is a reliable and valid instrument for assessing the levels of SRL support within learning environments. Additional implications for further research to support self-regulated learning are discussed.

Keywords

• Self-regulated learning
• SRL-S rubric
• Validity
• Reliability

References

1. Ameloot, E., Rotsaert, T., Ameloot, T., Rienties, B., & Schellens, T. (2024). Supporting students’ basic psychological needs and satisfaction in a blended learning environment through learning analytics. Computers & Education, 209, 104949.
2. American Educational Research Association. (2014). Standards for educational and psychological testing (AERA, APA, and NCME). Washington, USA: American Educational Research Association, ISBN 978-0-935302-35-6. Andrade, H., & Du, Y. (2007). Student responses to criteria-referenced self-assessment. Assessment & Evaluation in Higher Education, 32(2), 159–181.
3. Dong, X., Yuan, H., Xue, H., Li, Y., Jia, L., Chen, J., Shi, Y., & Zhang, X. (2024). Factors influencing college students’ self-regulated learning in online learning environment: A systematic review. Nurse Education Today, 133, 106071.
4. Goda, Y., Yamada, M., Matsuda, T., Kato, H., Saito, Y., & Miyagawa, H. (2022). From Adaptive Learning Support to Fading Out Support for Effective Self-Regulated Online Learning. Research Anthology on Remote Teaching and Learning and the Future of Online Education, 254–274.
5. Harris, J., Grandgenett, N., & Hofer, M. (2010). Testing a TPACK-Based Technology Integration Assessment Rubric. In D. Gibson & B. Dodge (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference 2010 (pp. 3833-3840). Chesapeake, VA: AACE
6. Heikkinen, S., Saqr, M., Malmberg. J., & Tedre, M. (2022). Supporting self-regulated learning with learning analytics interventions – a systematic literature review. Education and Information Technologies, 28, 3059–3088. https://doi.org/10.1007/s10639-022-11281-4
7. Järvelä, S., Nguyen, A., & Molenaar, I. (2023). Advancing SRL research with artificial intelligence. Computers in Human Behavior, 147, 107847.
8. Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130 144.

9. Lodge, J.M., & Harrison, W.J. (2019). The Role of Attention in Learning in the Digital Age. The Yale Journal of Biology and Medicine, 92(1), 21–28.
10. McGraw, K.O., & Wong, S.P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.
11. Mirriahi, N., Joksimović, S., Gašević, D., et al. (2018). Effects of instructional conditions and experience on student reflection: A video annotation study. Higher Education Research and Development, 37(6), 1245–59
12. Molenaar, I., Mooij, S.D., Azevedo, R., Bannert, M., Järvelä, S., & Gašević, D. (2023). Measuring self-regulated learning and the role of AI: Five years of research using multimodal multichannel data. Computers in Human Behavior, 139, 107540.
13. Moskal, M., & Leydens, J. (2019). Scoring Rubric Development: Validity and Reliability. Practical Assessment, Research, and Evaluation, 7, 10.
14. OECD (2019). Getting skills right: Future-ready adult learning systems. Paris: OECD Publishing.
15. Panadero, E. (2017). A Review of Self-regulated Learning: Six Models and Four Directions for Research. Front. Psychol, 8, 422.
16. Panadero, E., Klug, J., & Järvelä, S. (2016). Third wave of measurement in the self-regulated learning field: When measurement and intervention come hand in hand. Scandinavian Journal of Educational Research, 60(6), 723–735.
17. Pintrich, P.R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P.R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). Academic Press.
18. Radović, S. (2024). Is it only about technology? The interplay between educational technology, teaching practice, and students’ learning activities. Journal of Computers in Education, 11, 743–762.
19. Radović, S., & Seidel, N. (2024a). Introducing the SRL-S rubric for evaluating technology-enhanced learning environments for self-regulated learning. Accepted for publication in Innovative Higher Education Journal.
20. Radović, S., & Siedel, N. (2024b). Bridging learning science and learning analytics: Self-Regulation Learning support (SRL-S) rubric. 14th International Conference on Learning Analytics & Knowledge (LAK24). 18. – 22. 3. 2024, Kyoto, Japan.
21. Radović, S., Seidel, N., Haake, J.M., & Kasakowskij, R. (2024a). Analyzing students' self-assessment practice in a distance education environment: Student behavior, accuracy, and task-related characteristics. Journal of Computer Assisted Learning, 40(2), 654–666.
22. Radović, S., Seidel, N., Menze, D., & Kasakowskij, R. (2024b). Investigating the effects of different levels of students’ regulation support on learning process and outcome: In search of the optimal level of support for self-regulated learning. Computers & Education, 215, 105041.
23. Reddy, M., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 35(4), 435-448.
24. Sghir, N., Adadi, A., & Lahmer, M. (2022). Recent advances in Predictive Learning Analytics: A decade systematic review (2012–2022). Education and Information Technologies, 28(7), 8299–8333.
25. Thaler, N., Kazemi, E., & Huscher, C. (2009). Developing a rubric to assess student learning outcomes using a class assignment. Teaching of Psychology, 36(2), 113–116.
26. Wang, S., Christensen, C., Cui, W., Tong, R., Yarnall, L., Shear, L., & Feng, M. (2023). When adaptive learning is effective learning: comparison of an adaptive learning system to teacher-led instruction. Interactive Learning Environments, 31(2), 793–803.
27. Wu, T.T., Lee, H.Y., Li, P.H., Huang, C.N., & Huang, Y.M. (2023). Promoting Self-Regulation Progress and Knowledge Construction in Blended Learning via ChatGPT-Based Learning Aid. Journal of Educational Computing Research, 61(8), 3–31.
28. Zimmerman, B.J. (2000). Attaining self-regulation: a social cognitive perspective. In Handbook of Self-Regulation, eds M. Boekaerts, P.R. Pintrich, and M. Zeidner (San Diego, CA: Academic Press), 13–40. Altman, D.G. (1999). Practical statistics for medical research. New York, NY: Chapman & Hall/CRC Press.
29. Araka, E., Maina, E., Gitonga, R., & Oboko, R. (2020). Research trends in measurement and intervention tools for self-regulated learning for e-learning environments—systematic review (2008–2018). Research and Practice in Technology Enhanced Learning, 15(1).
30. Ceron, J., Baldiris, S., Quintero, J., Garcia, R.R., Saldarriaga, G.L.V., Graf, S., & De La Fuente Valentin, L. (2021). Self-Regulated Learning in Massive Online Open Courses: A State-of-the-Art Review. IEEE Access, 9, 511–528.
31. Devolder, A., Van Braak, J., & Tondeur, J. (2012). Supporting self‐regulated learning in computer‐based learning environments: systematic review of effects of scaffolding in the domain of science education. Journal of Computer Assisted Learning, 28(6), 557–573.
32. Edisherashvili, N., Saks, K., Pedaste, M., & Leijen, Ä. (2022). Supporting Self-Regulated Learning in Distance Learning Contexts at Higher Education Level: Systematic Literature Review. Front. Psychol., 12, 792422.
33. Gambo, Y., & Shakir, M.Z. (2021). Review on self-regulated learning in smart learning environment. Smart Learning Environments, 8(1), 12. https://doi.org/10.1186/s40561-021-00157-8
34. Garcia, R., Falkner, K., & Vivian, R. (2018). Systematic literature review: Self-Regulated Learning strategies using e-learning tools for Computer Science. Computers & Education, 123, 150–163.
35. Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness is not enough: Pitfalls of learning analytics dashboards in the educational practice. In European conference on technology enhanced learning (pp. 82-96). Springer, Cham.
36. Matcha, W., Uzir, N. A., Gašević, D., & Pardo. A. (2020). A Systematic Review of Empirical Studies on Learning Analytics Dashboards: A Self-Regulated Learning Perspective. IEEE Transactions on Learning Technologies, 13(2), 226-245.
37. Pérez-Álvarez, R., Maldonado-Mahauad, J., & Pérez-Sanagustín, M. (2018). Tools to Support Self-Regulated Learning in Online Environments: Literature Review. In Lecture notes in computer science (pp. 16–30).
38. Viberg, O., Khalil, M., & Baars, M. (2020). Self-regulated learning and learning analytics in online learning environments. In Proceedings of the 10th International Conference on Learning Analytics Knowledge (LAK’20). ACM, New York, NY, USA, 11 pages.