Adaptation of the Massive Open Online Courses (MOOCs) Engagement Scale into Turkish: Validity and Reliability Study

Abstract

The purpose of this study is to adapt and validate the Turkish version of the Massive Open Online Course Engagement Scale (MES) developed by Deng et al. (2020). The scale is divided into four sub-dimensions: behavioral, cognitive, affective, and social engagement. The scale form with language equivalence was applied to 221 students in the Education Faculty who enrolled in MOOCs and completed at least one course for validity and reliability research. The CFA examined the adapted scale's construct validity in the sample. As a result of CFA, 12 items, 4-factors structure of the scale was preserved in Turkish MOOCs. According to the findings, the model had an acceptable fit with the data (χ2 =131.8, df=48, p<.01, CFI=0.95, RMSEA=0.08, TLI=0.93, SRMR=0.08). The first-level CFA revealed the scale's 4-factor structure of behavioral, cognitive, affective, and social engagement. The combined reliability coefficient of the scale varied between .84 and .89 in each sub-dimension. Consequently, it was determined that the measurements were sufficiently reliable. According to the findings, the adapted Turkish version of the original version of the scale is compatible with Turkish culture and can be utilized in academic researches.

Keywords

• Massive Open Online Courses
• MOOCs
• Distance Education
• Learner Engagement
• Student Engagement

Kitlesel Açık Çevrimiçi Dersler (KAÇD) Bağlılık Ölçeği'nin Türkçeye Uyarlanması, Geçerlik ve Güvenirlik Çalışması

Öz

Bu çalışmanın amacı, Deng ve diğerleri (2020) tarafından geliştirilen Kitlesel Açık Çevrimiçi Kurs Bağlılığı Ölçeği'nin (MES) Türkçe uyarlamasını ve geçerliliğini araştırmaktır. Ölçek davranışsal, bilişsel, duyuşsal ve sosyal bağlılık olmak üzere dört alt boyuta ayrılmıştır. Dil eşdeğerliği sağlanan ölçek formu, geçerlilik ve güvenilirlik araştırması için MOOC'lara kaydolan ve en az bir kursu tamamlayan Eğitim Fakültesi'ndeki 221 öğrenciye uygulanmıştır. Uyarlanan ölçeğin örneklemdeki yapı geçerliliğini doğrulayıcı faktör analizi (DFA) ile incelemiştir. DFA sonucunda, ölçeğin 12 maddeli, 4 faktörlü yapısının Türk MOOC'larında korunduğu görülmüştür. Bulgulara göre, model veri ile kabul edilebilir bir uyum göstermiştir (χ2 =131,8, df=48, p<,01, CFI=0,95, RMSEA=0,08, TLI=0,93, SRMR=0,08). Birinci düzey DFA bulguları ölçeğin davranışsal, bilişsel, duygusal ve sosyal bağlılıktan oluşan 4 faktörlü yapısının korunduğunu ortaya koymuştur. Ölçeğin birleşik güvenirlik katsayısı her bir alt boyutta ,84 ile ,89 arasında değişmektedir. Sonuç olarak, ölçümlerin yeterince güvenilir olduğu tespit edilmiştir. Elde edilen bulgulara göre, ölçeğin orijinal versiyonunun Türkçeye uyarlanmış hali Türk kültürüyle uyumludur ve akademik çalışmalarda kullanılabilir.

Anahtar Kelimeler

• Kitlesel Açık Çevrimiçi Dersler
• KAÇD
• MOOCs
• Uzaktan Eğitim
• Öğrenen Bağlılığı
• Öğrenci Bağlılığı

Kaynakça

1. Abelson, H. (2008). The creation of OpenCourseWare at MIT. Journal of Science Education and Technology, 17(2), 164-174. https://doi.org/10.1007/s10956-007-9060-8
2. Alarcón, D., & Sánchez, J. A. (2015). Assessing convergent and discriminant validity in the ADHD-R IV rating scale: User-written commands for Average Variance Extracted (AVE), Composite Reliability (CR), and Heterotrait-Monotrait ratio of correlations (HTMT). https://www.stata.com/meeting/spain15/abstracts/materials/spain15_alarcon.pdf
3. Albelbisi, N. A. (2020). Development and validation of the MOOC success scale (MOOC-SS). Education and Information Technologies, 25, 4535-4555. https://doi.org/10.1007/s10639-020-10186-4
4. Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of school psychology, 44(5), 427-445. https://doi.org/10.1080/00091383.2011.533096
5. Axelson, R. D., & Flick, A. (2010). Defining student engagement. Change: The magazine of higher learning, 43(1), 38-43. https://doi.org/10.1080/00091383.2011.533096b
6. Aydın, I. E., & Yazıcı, M. (2020). Drop-Out in MOOCs. Turkish Online Journal of Educational Technology-TOJET, 19(3), 9-17.
7. Bentler, P.M. (1980). Multivariate analysis with latent variables: Causal modeling. Annual review of psychology, 31, 419-456. https://doi.org/10.1146/annurev.ps.31.020180.002223b
8. Bentler, P.M., & Bonett, D.G. (1980). Significance Tests and Goodness of Fit in the Analysis of Covariance Structures. Psychological Bulletin, 88, 588-606. https://doi.org/10.1037/0033-2909.88.3.58b

9. Biçer, E. ve Reisoğlu, İ. (2022). Mooc öğrenci meşguliyeti ölçeğinin Türkçe’ye uyarlanması: Geçerlik ve güvenirlik çalışması. Eğitim Teknolojisi Kuram ve Uygulama, 12(1), 1-20. https://doi.org/10.17943/etku.910355b
10. Bingham, G. E., & Okagaki, L. (2012). Ethnicity and student engagement. In Handbook of research on student engagement (65-95). Springer. https://doi.org/10.1007/978-1-4614-2018-7_4.
11. Boomsma, A., & Hoogland, J. J. (2001). The Robustness of LISREL Modeling Revisited. In R. Cudeck, S. du Toit, & D. Sörbom (Eds.), Structural Equation Models: Present and Future. A Festschrift in Honor of Karl Jöreskog (pp. 139-168). Scientific Software International. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.11.684&rep=rep1&type=pdf
12. Bozkurt, A., Koçdar, S., Çağıltay, K., Eşfer, S., Çelik, B., Karaman, S., ve Kurşun, E. (2021). Türkiye’de kitlesel açık çevrimiçi dersler (KAÇD) ve Türk yükseköğretimi bağlamında bir değerlendirme. Yükseköğretim Dergisi, 11(2 Pt 2), 521–536. https://doi.org/10.2399/yod.20.702064.
13. Brown, M., & Cudeck, R. (1993). Alternative ways of assessing model fit. In: K. Bollen & J. Long, (Eds), Testing Structural Equation Models (pp. 136–162). Sage Publications.
14. Buelow, J. R., Barry, T., & Rich, L. E. (2018). Supporting learning engagement with online students. Online Learning, 22(4), 313–340. http://files.eric.ed.gov/fulltext/EJ1202338.pdf
15. Burch, G. F., Heller, N. A., Burch, J. J., Freed, R., & Steed, S. A. (2015). Student engagement: Developing a conceptual framework and survey instrument. Journal of Education for Business, 90(4), 224–229. https://doi.org/10.1080/08832323.2015.1019821.
16. Byrne, B. M. (2010). Structural equation modeling with AMOS: Basic concepts, applications, and programming (2nd ed.). Routledge/Taylor & Francis Group.
17. Christenson, S. L., Stout, K., & Pohl, A. (2012). Check & Connect: A comprehensive student engagement intervention: Implementing with fidelity. The University of Minnesota, Institute on Community Integration. http://checkandconnect.umn.edu/manual/default.html
18. Connell, J. P., Spencer, M. B. ve Aber, J. L. (1994). Educational risk and resilience in African-American youth: Context, self, action, and outcomes in school. Child Development, 65, 493-506. https://doi.org/10.2307/1131398
19. Cormier, D., ve Siemens, G. (2010). Through the open door: Open courses as research, learning, and engagement. EDUCAUSE Review, 45(4), 30-39. https://www.islandscholar.ca/islandora/object/ir%3A3675/datastream/PDF/view
20. Daniels, L. M., Adams, C., ve McCaffrey, A. (2016). Emotional and social engagement in a Massive Open Online Course: An examination of Dino 101. In Emotions, technology, and learning, 25-41. Academic Press. https://doi.org/10.1016/B978-0-12-800649-8.00004-3.
21. de Barba, P., Kennedy, G. E., & Ainley, M. D. (2016). The role of students' motivation and participation in predicting performance in a MOOC. Journal of Computer Assisted Learning, 32(3), 218–231. https://doi.org/10.1111/jcal.12130.
22. De Freitas, S. I., Morgan, J., & Gibson, D. (2015). Will MOOCs transform learning and teaching in higher education? Engagement and course retention in online learning provision. British journal of educational technology, 46(3), 455-471. https://doi.org/10.1111/bjet.12268.
23. Deng, R., Benckendorff, P., Gannaway, D. (2019). Progress and new directions for teaching and learning in MOOCs. Comput. Educ.,129, 48–60. https://doi.org/10.1016/j.compedu.2018.10.019.
24. Deng, R., Benckendorff, P., Gannaway, D. (2020). Learner engagement in MOOCs: Scale development and validation. British Journal of Educational Technology, 51(1), 245-262. https://doi.org/10.1111/bjet.12810.
25. Deng, R. (2021). Emotionally Engaged learners are more satisfied with online courses. Sustainability, 13(20), 11169. https://doi.org/10.3390/su132011169.
26. Dixson, M. D. (2012). Creating effective student engagement in online courses: What do students find engaging?. Journal of the Scholarship of Teaching and Learning, 10(2), 1–13. https://scholarworks.iu.edu/journals/index.php/josotl/article/view/1744.
27. Doğan, U. (2014). Validity and Reliability of Student Engagement Scale. Bartın Üniversitesi Eğitim Fakültesi Dergisi, 3(2), 390-403. https://doi.org/10.14686/BUEFAD.201428190
28. Downes, S. (2011). Connectivism and Connective Knowledge. https://www.downes.ca/cgi-bin/page.cgi?post=54540
29. Ergün, E. ve Usluel, Y. K. (2015). Çevrimiçi öğrenme ortamlarında öğrenci bağlılık ölçeği’nin Türkçe uyarlaması: geçerlik ve güvenirlik çalışması. Eğitim Teknolojisi Kuram ve Uygulama, 5(1), 20-33. https://doi.org/10.17943/etku.64661.
30. Estrada-Molina, O., & Fuentes-Cancell, D. R. (2022). Engagement and desertion in MOOCs: Systematic review. Comunicar, 30(70), 111-124. https://files.eric.ed.gov/fulltext/EJ1320958.pdf
31. Finn, J. D., & Zimmer, K. S. (2012). Student engagement: What is it? Why does it matter?. In Handbook of research on student engagement (97-131). Springer. https://doi.org/10.1007/978-1-4614-2018-7_5.
32. Fletcher, Adam. (2015). “Defining Student Engagement: A Literature Review.” Soundout: Promoting Meaningful Student Involvement, Student Voice and Student Engagement. https://soundout.org/defining-student-engagement-a-literature-review.
33. Fornell, C. & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of marketing research, 39-50. https://doi.org/10.1177/002224378101800104.
34. Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of educational research, 74(1), 59-109. https://doi.org/10.3102/00346543074001059.
35. Glass, C. R., Shiokawa‐Baklan, M. S., & Saltarelli, A. J. (2016). Who takes MOOCs?. New Directions for Institutional Research, 2015(167), 41-55. https://doi.org/10.1002/ir.20153.
36. Goodwin, L. D. (1999). The role of factor analysis in the estimation of construct validity. Measurement in Physical Education and Exercise Science, 3(2), 85-100. https://doi.org/10.1207/s15327841mpee0302_2
37. Greene, B. A., Miller, R. B., Crowson, H. M., Duke, B. L., & Akey, K. L. (2004). Predicting high school students' cognitive engagement and achievement: Contributions of classroom perceptions and motivation. Contemporary educational psychology, 29(4), 462-482. https://doi.org/10.1016/j.cedpsych.2004.01.006.
38. Groccia, J. E. (2018). What is student engagement?. New directions for teaching and learning, 2018(154), 11-20. https://doi.org/10.1002/tl.20287.
39. Gün, F., Turabik, T., Arastaman, G., ve Akbaşlı, S. (2019). Üniversite Öğrencilerinin Okul Bağlılığı Ölçeğinin Türk kültürüne uyarlanması: Geçerlik ve güvenirlik çalışması. İnönü Üniversitesi Eğitim Fakültesi Dergisi, 20(2), 507-520. https://doi.org/10.17679/inuefd.457919.
40. Gününç, S. (2014). The relationships between student engagement and their academic achievement. International Journal on New Trends in Education and their implications, 5(4), 216-231.
41. Günüç, S. ve Kuzu, A. (2015). Student engagement scale: development, reliability, and validity. Assessment & Evaluation in Higher Education, 40(4), 587-610. https://doi.org/10.1080/02602938.2014.938019.
42. Haggard, S., Wang, B. H., & He, X. L. (2014). The MOOC is maturing. Educational Research, 5(412), 92-112.
43. Halverson, L. R., & Graham, C. R. (2019). Learner engagement in blended learning environments: A conceptual framework. Online Learning, 23(2), 145-178. https://doi.org/doi:10.24059/olj.v23i2.1481.
44. Hambleton, R.K., & Kanjee, A. (1993, April). Enhancing the Validity of Cross-Cultural Studies: Improvements in Instrument Translation Methods. Paper presented at the Annual Meeting of the American Educational Research Association, GA. http://files.eric.ed.gov/fulltext/ED362537.pdf
45. Haymana, B., ve Dağhan, G. (2020). Investigation of research about Massive Open Online Courses (moocs): A Thematic content analysis study. Journal of Computer and Education Research, 8(16), 787-820. https://doi.org/10.18009/jcer.772010.
46. Hew, K. F. (2016). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCS. British Journal of Educational Technology, 47(2), 320-341. https://doi.org/10.1111/bjet.12235.
47. Goopio, Joselyn ve Cheung, Catherine (2021). The MOOC dropout phenomenon and retention strategies, Journal of Teaching in Travel & Tourism, 21:2, 177-197, https://doi.org/10.1080/15313220.2020.1809050.
48. İlhan, M. ve Çetin, B. (2013). Çevrimiçi öğrenmeye yönelik hazır bulunuşluk ölçeği'nin (ÇÖHBÖ) türkçe formunun geçerlik ve güvenirlik çalışması . Eğitim Teknolojisi Kuram ve Uygulama, 3 (2), 72-101. https://dergipark.org.tr/tr/download/article-file/71808
49. Jimerson, S. R., Campos, E., & Greif, J. L. (2003). Toward an understanding of definitions and measures of school engagement and related terms. The California School Psychologist, 8(1), 7–27. https://doi.org/10.1007/BF03340893.
50. Jordan, K. (2014). Initial trends in enrolment and completion of massive open online courses. Int. Rev. Res. Open Dist. Learn. 15, 1651. https://doi.org/10.19173/irrodl.v15i1.1651.
51. Karasar, N. (2012). Bilimsel araştırma yöntemi. Nobel Yayıncılık.
52. Kline, R.B. (2015). Principles and Practice of Structural Equation Modeling. Guilford Press.
53. Koller, D., Ng, A., Do, C., & Chen, Z. (2013). Retention and intention in massive open online courses: In depth. Educause review, 48(3), 62-63.
54. Kuh, G. D. (2003). What we're learning about student engagement from NSSE: Benchmarks for effective educational practices. Change: The magazine of higher learning, 35(2), 24-32. https://doi.org/10.1080/00091380309604090.
55. Kuh, G. D. (2009). The national survey of student engagement: Conceptual and empirical foundations. New directions for institutional research, 141, 5-20. https://doi.org/10.1002/ir.283.
56. Malhotra, N. K. (2010). Marketing research: An applied orientation (6th ed.). Upper Saddle River. Pearson.
57. Marelize Malan (2020) Engaging students in a fully online accounting degree: an action research study, Accounting Education, 294, 321-339. https://doi.org/10.1080/09639284.2020.1787855.
58. Marks, H.M. (2000). Student Engagement in Instructional Activity: Patterns in the Elementary, Middle, and High School Years. American Educational Research Journal, 37, 153-184. https://doi.org/10.3102/00028312037001153.
59. Maroco, J., Maroco, A. L., Campos, J. A. D. B., & Fredricks, J. A. (2016). University student’s engagement: development of the University Student Engagement Inventory (USEI). Psicologia: Reflexão e Crítica, 29. https://doi.org/10.1186/s41155-016-0042-8.
60. Moskal, P., Thompson, K., & Futch, L. (2015). Enrollment, engagement, and satisfaction in the BlendKit faculty development open, online course. Online Learning, 19(4). https://files.eric.ed.gov/fulltext/EJ1079573.pdf
61. Muthén, L. K., & Muthén, B. O. (1998). Mplus user’s guide. Eight Edition. Muthén & Muthén. https://www.statmodel.com/download/usersguide/MplusUserGuideVer_8.pdf
62. Newmann, F. M. (1992). Student engagement and achievement in American secondary schools. Teachers College Press.
63. Önen, Emine (2014). Öğrencinin okula bağlılığı ölçeği: Türk ortaokul ve lise öğrencileri için uyarlama çalışması. PDR Türk Psikolojik Danışma ve Rehberlik Dergisi, 5(42).
64. Öner, N. (1997) Türkiye’de Kullanılan Psikolojik Testler, 3. Basım, 458, Boğaziçi Üniversitesi Yayınları, İstanbul.
65. Pintrich, P. R., & Schrauben, B. (1992). Students’ motivational beliefs and their cognitive engagement in classroom academic tasks. Student perceptions in the classroom, 7, 149-183.
66. Ramesh, A., Goldwasser, D., Huang, B., Daumé III, H., & Getoor, L. (2013, December). Modeling learner engagement in MOOCs using probabilistic soft logic. In NIPS workshop on data-driven education. 21, 62. https://linqs.github.io/linqs-website/assets/resources/ramesh-dde13.pdf
67. Redmond, P., Heffernan, A., Abawi, L., Brown, A., & Henderson, R. (2018). An online engagement framework for higher education. Online Learning, 22(1), 183–204. https://doi.org/10.24059/olj.v22i1.1175
68. Richardson, J. C., & Newby, T. (2006). The role of students' cognitive engagement in online learning. American Journal of Distance Education, 20(1), 23-37. https://doi.org/10.1207/s15389286ajde2001_3
69. Rotermund, S. L. (2010). The role of psychological precursors and student engagement in a process model of high school dropout. University of California, Santa Barbara.
70. Rothkrantz, L. (2016, April). Dropout rates of regular courses and MOOCs. In International Conference on Computer Supported Education (pp. 25-46). Springer, Cham.
71. Schaffhause, Dian. (2021). MOOC Enrollment Explodes in 2020. https://campustechnology.com/Articles/2021/01/05/MOOC-Enrollment-Explodes-in-2020
72. Shah, D. (2021a). MOOC Roundup 2021. Class Central. https://www.classcentral.com/report/tag/mooc-roundup-2021/
73. Shah, D. (2021b). A Decade of MOOCs: A Review of MOOC Stats and Trends in 2021. Class Central. https://www.classcentral.com/report/moocs-stats-and-trends-2021/
74. Steve Carson (2009) The unwalled garden: growth of the OpenCourseWare Consortium, 2001–2008, Open Learning: The Journal of Open, Distance and e-Learning, 24(1), 23-29, https://doi.org/10.1080/02680510802627787.
75. Stevens, J. (2002). Applied multivariate statistics for the social sciences. Mahwah, Lawrence Erlbaurn Associates.
76. Stöhr, C., Stathakarou, N., Mueller, F., Nifakos, S. & McGrath, C. (2019). Videos as learning objects in MOOCs: A study of specialist and non‐specialist participants' video activity in MOOCs. British Journal of Educational Technology, 50(1), 166-176. https://doi.org/10.1111/bjet.12623.
77. Stracke, C. M., Downes, S., Conole, G., Burgos, D., & Nascimbeni, F. (2019). Are MOOCs Open Educational Resources? A Literature Review on History, Definitions, and Typologies of OER and MOOCs. Open Praxis, 11(4), 331-341. https://files.eric.ed.gov/fulltext/EJ1251318.pdf
78. Sun, J. C.-Y. ve Rueda, R. (2012). Situational interest, computer self-efficacy, and self-regulation: their impact on student engagement in distance education. British Journal of Educational Technology, 43(2), 191-204. https://doi.org/10.1111/j.1467-8535.2010.01157.x.
79. Sun, Y., Guo, Y., & Zhao, Y. (2020). Understanding the determinants of learner engagement in MOOCs: An adaptive structuration perspective. Computers & Education, 157, https://doi.org/10.1016/j.compedu.2020.103963.
80. Sun, Y., Ni, L., Zhao, Y., Shen, X.-L. & Wang, N. (2019), Understanding students’ engagement in MOOCs: An integration of self-determination theory and theory of relationship quality. British Journal of Educational Technology, 50(6). 3156-3174. https://doi.org/10.1111/bjet.12724.
81. Topal, M., İstanbullu, A., & Akgün, Ö. (2020). Çevrimiçi öğrenmede öğrenci bağlılığı ölçeği üniversite öğrencisi formunun psikometrik özellikleri. International Journal of Human Sciences, 17(1). 104-116. https://doi.org/10.14687/jhs.v17i1.5698.
82. Trowler, V. (2010). Student engagement literature review. The higher education academy, 11(1), 1-15. https://www.academia.edu/download/30908577/StudentEngagementLiteratureReview.pdf
83. Uğur, E. ve Akın, A. (2015). Öğrenci bağlılığı ölçeği Türkçe formunun geçerlik ve güvenirlik çalışması. SDU International Journal of Educational Studies, 2(1), 53-59. https://dergipark.org.tr/en/download/article-file/195336.
84. Veletsianos, G., Collier, A., & Schneider, E. (2015). Digging deeper into learners' experiences in MOOCs: Participation in social networks outside of MOOC s, notetaking, and contexts surrounding content consumption. British Journal of Educational Technology, 46(3), 570-587. https://doi.org/10.1111/bjet.12297.
85. Wang R, Cao J, Xu Y and Li Y (2022) Learning engagement in massive open online courses: A systematic review. Front. Educ. 7:1074435. https://doi.org/10.3389/feduc.2022.1074435.
86. Wang, X., Tan, S. C., & Li, L. (2020). Measuring university students’ technostress in technology-enhanced learning: Scale development and validation. Australasian Journal of Educational Technology, 36(4), 96-112. https://doi.org/10.14742/ajet.5329.
87. Watted, A., & Barak, M. (2018). Motivating factors of MOOC completers: Comparing between university-affiliated students and general participants. The Internet and Higher Education, 37, 11–20. https://doi.org/10.1016/j.iheduc.2017.12.001.
88. Weston, R., & Gore Jr, P. A. (2006). A brief guide to structural equation modeling. The counseling psychologist, 34(5), 719-751. https://doi.org/10.1177/0011000006286345.
89. Xiong, Y., Li, H., Kornhaber, M. L., Suen, H. K., Pursel, B., & Goins, D. D. (2015). Examining the relations among student motivation, engagement, and retention in a MOOC: A structural equation modeling approach. Global Education Review, 2(3), 23-33. https://files.eric.ed.gov/fulltext/EJ1074099.pdf
90. Vardi, M. Y. (2012). Will MOOCs destroy academia?. Communications of the ACM, 55(11), 5-5. https://doi.org/10.1145/2366316.2366317.
91. Yafei Liu, Mingli Zhang, Dan Qi & Yan Zhang (2022) Understanding the role of learner engagement in determining MOOCs satisfaction: a self-determination theory perspective, Interactive Learning Environments. 1-15. https://doi.org/10.1080/10494820.2022.2028853.
92. Yang, Q., & Lee, Y. C. (2021). The critical factors of student performance in MOOCs for sustainable education: a case of Chinese universities. Sustainability, 13(14), 8089.