IDENTIFYING LEARNERS’ EXPECTATIONS FROM LEARNING ANALYTICS DASHBOARDS IN THE CONTEXT OF ANALYTICS TYPES

Abstract

This case study aims to discover elements that learners expect from learning analytics dashboards and propose design principles based on these expectations. Focus group interviews were conducted with 20 graduate students with previous e-learning experience to inform design principles for learning analytics dashboards. Interviews were conducted with 5 groups, lasting an average of 53 minutes. The gathered information was analyzed through content analysis. The research findings were divided into five themes: four were expectations for information within the scope of learning analytics types (descriptive analytics, diagnostic analytics, predictive analytics, predictive analytics), and one theme was generic expectations for how the learning analytics dashboard should be designed. For descriptive analytics, respondents indicated that they would like to see information on how they were performing in relation to their learning objectives, as well as how their performance compared to the group/class average. For diagnostic analytics, respondents indicated that they would like to see information about learning deficiencies, performance anomalies, as well as the relation between time spent on a topic and performance. Although estimations of success within the context of predictive analytics are widely expected, information on how to follow a path to be successful within the context of prescriptive analytics is also expected. The study also presented the general expectations of learners from the learning analytics dashboards. Finally, design principles for learning analytics dashboards in the context of learning analytics types are presented.

Keywords

• learning analytics dashboards
• descriptive analytics
• diagnostic analytics
• predictive analytics
• prescriptive analytics

ÖĞRENME ANALİTİĞİ DÜZEYLERİ BAĞLAMINDA ÖĞRENME PANELİ TASARIMINA YÖNELİK ÖĞRENEN BEKLENTİLERİNİN BELİRLENMESİ

Öz

Bu çalışma, öğrenenlerin öğrenme analitiği düzeyleri bağlamında öğrenme panelinde yer almasını bekledikleri öğeleri keşfetmeyi ve buna uygun tasarım ilkeleri ortaya koymayı amaçlayan bir durum çalışmasıdır. Bu kapsamda daha önce e-öğrenme deneyimi olan 20 lisansüstü öğrencisiyle odak grup görüşmeleri gerçekleştirilmiştir. Odak grup görüşmeleri 5 farklı oturumda gerçekleştirilmiş ve her oturum ortalama 53 dakika sürmüştür. Görüşmelerden elde edilen veriler içerik analizi yöntemiyle çözümlenmiştir. Araştırma sonucunda elde edilen bulgular; dördü öğrenme analitiği düzeyleri (betimleyici analitikler, tanılayıcı analitikler, yordayıcı analitikler, öngörü analitikleri) kapsamında öğrenme panelinde yer alması gereken bilgilere yönelik beklentiler, biri ise bu bilgilerin öğrenme panelinde ne şekilde organize edilip sunulacağına ilişkin beklentiler olmak üzere beş alt başlık altında analiz edilip yorumlanmıştır. Katılımcılar betimleyici analitikler kapsamında öğrenme hedeflerine göre ne durumda olduklarına, gruba/sınıfa göre performanslarının nasıl olduğuna ilişkin bilgiler görmek istediklerini belirtmişlerdir. Tanılayıcı analitikler kapsamında ise katılımcılar öğrenme eksikliklerinin tespiti, performanslarındaki değişimlerin saptanması ve performans ile harcanan zaman ilişkisinin gösterimi ile ilgili bilgileri görmek istediklerini ifade etmişlerdir. Yordayıcı analitikler kapsamında başarı kestirimlerinin sunulması yaygın olarak beklenirken öngörü analitikleri kapsamında buna ek olarak başarılı olmak için nasıl bir yol izlemesi gerektiğine ilişkin bilgiler sunulması beklenmiştir. Çalışmada ayrıca öğrenme analitiği düzeylerinden bağımsız olarak öğrenenlerin öğrenme paneli tasarımına yönelik genel beklentileri sunulmuştur. Son olarak öğrenme analitiği düzeyleri bağlamında öğrenme panelinin tasarımına yönelik tasarım ilkeleri sunulmuştur.

Anahtar Kelimeler

• öğrenme paneli
• betimleyici analitikler
• tanılayıcı analitikler
• yordayıcı analitikler
• öngörü analitikleri

References

1. Aldowah, H., Al-Samarraie, H., & Fauzy, W. M. (2019). Educational data mining and learning analytics for 21st century higher education: A review and synthesis. Telematics and Informatics, 37, 13-49.
2. Brock, T. R. (2017). Performance analytics: the missing big data link between learning analytics and business analytics. Performance Improvement, 56(7), 6-16.
3. Brown, A., & Green, T. (2018). Issues and trends in instructional technology: consistent growth in online learning, digital content, and the use of mobile technologies. In Educational media and technology yearbook (pp. 61-71). Springer, Cham.
4. Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. EDUCAUSE review, 42(4), 40.
5. Chatti, M. A., Dyckhoff, A. L., Schroeder, U., & Thüs, H. (2012). A reference model for learning analytics. International Journal of Technology Enhanced Learning, 4(5-6), 318-331.
6. Conole, G., & Alevizou, P. (2010). A literature review of the use of Web 2.0 tools in Higher Education. A report commissioned by the Higher Education Academy.
7. Cuban, L. (1986). Teachers and machines: The classroom use of technology since 1920. Teachers College Press.
8. Delen, D., & Ram, S. (2018). Research challenges and opportunities in business analytics. Journal of Business Analytics, 1(1), 2-12.

9. Deshpande, P. S., Sharma, S. C., & Peddoju, S. K. (2019). Predictive and prescriptive analytics in big-data era. In Security and data storage aspect in cloud computing (pp. 71-81). Springer, Singapore.
10. Ellis, C. (2013). Broadening the scope and increasing the usefulness of learning analytics: The case for assessment analytics. British Journal of Educational Technology, 44(4), 662-664.
11. Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64-71.
12. Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., & Hlosta, M. (2019). A large-scale implementation of predictive learning analytics in higher education: The teachers’ role and perspective. Educational Technology Research and Development, 67(5), 1273-1306.
13. Herodotou, C., Rienties, B., Hlosta, M., Boroowa, A., Mangafa, C., & Zdrahal, Z. (2020). The scalable implementation of predictive learning analytics at a distance learning university: Insights from a longitudinal case study. The Internet and Higher Education, 45, 100725.
14. Hilliger, I., Ortiz-Rojas, M., Pesántez-Cabrera, P., Scheihing, E., Tsai, Y. S., Muñoz-Merino, P. J., ... & Pérez-Sanagustín, M. (2020). Identifying needs for learning analytics adoption in Latin American universities: A mixed-methods approach. The Internet and Higher Education, 45, 100726.
15. Hindle, G., Kunc, M., Mortensen, M., Oztekin, A., & Vidgen, R. (2020). Business analytics: Defining the field and identifying a research agenda. European Journal of Operational Research, 281(3), 483-490.
16. Hsu, Y. C., Hung, J. L., & Ching, Y. H. (2013). Trends of educational technology research: More than a decade of international research in six SSCI-indexed refereed journals. Educational Technology Research and Development, 61(4), 685-705.
17. Ifenthaler, D. (2017). Are higher education institutions prepared for learning analytics?. TechTrends, 61(4), 366-371.
18. Ifenthaler, D., & Yau, J. Y. K. (2020). Utilising learning analytics to support study success in higher education: a systematic review. Educational Technology Research and Development, 68(4), 1961-1990.
19. Januszewski, A., & Molenda, M. (2008). Chapter 1: Definition. Educational technology: A definition with commentary. Lawrence Erlbaum Associates.
20. Jo, I. H., Yu, T., Lee, H., & Kim, Y. (2015). Relations between student online learning behavior and academic achievement in higher education: A learning analytics approach. In Emerging issues in smart learning (pp. 275-287). Springer, Berlin, Heidelberg.
21. Kimmons, R. (2020). Current trends (and missing links) in educational technology research and practice. TechTrends, 64(6), 803-809.
22. Lai, J. W., & Bower, M. (2020). Evaluation of technology use in education: Findings from a critical analysis of systematic literature review
23. Lim, L. A., Gentili, S., Pardo, A., Kovanović, V., Whitelock-Wainwright, A., Gašević, D., & Dawson, S. (2021). What changes, and for whom? A study of the impact of learning analytics-based process feedback in a large course. Learning and Instruction, 72, 101202.
24. Nouira, A., Cheniti‐Belcadhi, L., & Braham, R. (2019). An ontology‐based framework of assessment analytics for massive learning. Computer Applications in Engineering Education, 27(6), 1343-1360.
25. Olson, T. M., & Wisher, R. A. (2002). The effectiveness of web-based instruction: An initial inquiry. International Review of Research in Open and Distributed Learning, 3(2), 1-17.
26. Pardo, A., Jovanovic, J., Dawson, S., Gašević, D., & Mirriahi, N. (2019). Using learning analytics to scale the provision of personalized feedback. British Journal of Educational Technology, 50(1), 128-138.
27. Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action Evaluation Framework: A Review of Evidence-Based Learning Analytics Interventions at the Open University UK. Journal of Interactive Media in Education, 2016(1).
28. Romero, C., & Ventura, S. (2020). Educational data mining and learning analytics: An updated survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(3), e1355.
29. Schumacher, C., & Ifenthaler, D. (2018). Features students really expect from learning analytics. Computers in human behavior, 78, 397-407.
30. Sedrakyan, G., Malmberg, J., Verbert, K., Järvelä, S., & Kirschner, P. A. (2020). Linking learning behavior analytics and learning science concepts: Designing a learning analytics dashboard for feedback to support learning regulation. Computers in Human Behavior, 107, 105512.
31. Sergis, S., & Sampson, D. G. (2016). School analytics: A framework for supporting school complexity leadership. In Competencies in teaching, learning and educational leadership in the digital age (pp. 79-122). Springer, Cham.
32. Shum, S. B., & Ferguson, R. (2012). Social learning analytics. Journal of educational technology & society, 15(3), 3-26.
33. Siemens, G., & Gasevic, D. (2012). Guest editorial-learning and knowledge analytics. Journal of Educational Technology & Society, 15(3), 1-2.
34. Susnjak, T., Ramaswami, G. S., & Mathrani, A. (2022). Learning analytics dashboard: a tool for providing actionable insights to learners. International Journal of Educational Technology in Higher Education, 19(1), 1-23.
35. Şahin, M., & Yurdugül, H. (2020). Educational data mining and learning analytics: past, present and future. Bartın University Journal of Faculty of Education, 9(1), 121-131.
36. Valle, N., Antonenko, P., Valle, D., Dawson, K., Huggins-Manley, A. C., & Baiser, B. (2021). The influence of task-value scaffolding in a predictive learning analytics dashboard on learners' statistics anxiety, motivation, and performance. Computers & Education, 173, 104288.
37. Viberg, O., Khalil, M., & Baars, M. (2020, Mart). Self-regulated learning and learning analytics in online learning environments: A review of empirical research. In Proceedings of the tenth international conference on learning analytics & knowledge (pp. 524-533).
38. Viberg, O., Engström, L., Saqr, M., & Hrastinski, S. (2022). Exploring students’ expectations of learning analytics: A person-centered approach. Education and Information Technologies, 1-21.
39. West, D., Luzeckyj, A., Toohey, D., Vanderlelie, J., & Searle, B. (2020). Do academics and university administrators really know better? The ethics of positioning student perspectives in learning analytics. Australasian Journal of Educational Technology, 36(2), 60-70.
40. Whitelock‐Wainwright, A., Gašević, D., Tejeiro, R., Tsai, Y. S., & Bennett, K. (2019). The student expectations of learning analytics questionnaire. Journal of Computer Assisted Learning, 35(5), 633-666.
41. Karaoglan Yilmaz, F. G., & Yilmaz, R. (2020). Student opinions about personalized recommendation and feedback based on learning analytics. Technology, Knowledge and Learning, 25(4), 753-768.
42. Yunita, A., Santoso, H. B., & Hasibuan, Z. A. (2021, Haziran). Research review on big data usage for learning analytics and educational data mining: A way forward to develop an intelligent automation system. In Journal of Physics: Conference Series (Vol. 1898, No. 1, p. 012044). IOP Publishing.