Systematic Mapping Study 2012-2017: Quality and Effectiveness Measurement in MOOC

Year 2019, Volume: 20 Issue: 1, 223 - 247, 01.01.2019

Rocio Quılıano-terreros Darinka Ramırez-hernandez Pablo Barnıol

https://doi.org/10.17718/tojde.522719

Cited By: 3

Abstract

Massive Open Online Courses (MOOCs) concentrate an important number of students and are set up as an alternative for acquiring knowledge and continuing education. The objective of this article is to analyze how the quality and effectiveness of the MOOCs were measured in empirical studies between 2012 and 2017. For which a systematic mapping study of articles was performed by using databases of Scopus and Web of Science. These articles were analyzed according to the: (1) context of the publication, (2) type of developing institutions and distribution platforms of the MOOC, and (3) characteristics of the empirical studies. It was identified that 54% of the developing institutions were universities and that 31% of the platforms that distributed the MOOC are of Coursera. These articles were also analyzed by type of study, the more frequently used indicators to measure the effectiveness, and the more frequently used focus points (approaches) to measure quality. The results of this study are useful because it allows having a general view of the most frequently utilized methodologies to measure quality and effectiveness in MOOCs.

Keywords

Systematic Mapping Study, Massive Open Online Courses, MOOCs, quality, effectiveness, measurement

References

• Adam, M., Young-Wolff, K. C., Konar, E., & Winkleby, M. (2015). Massive open online nutrition and cooking course for improved eating behaviors and meal composition. International Journal of Behavioral Nutrition and Physical Activity, 12, 143. Alcock, S. E., Dufton, J. A., & Durusu-Tanriover, M. (2016). Archaeology and the MOOC: Massive, open, online, and opportunistic. Journal of Social Archaeology, 16(1), 3-31. Aleman De la Garza, L., Sancho-Vinuesa, T., & Gómez Zermeño, M. (2015). Indicators of pedagogical quality for the design of a massive open online course for teacher training. Universities and Knowledge Society Journal, 12(1), 104-118. Arias Masa, J. (2008). Evaluación de la calidad de Cursos Virtuales: Indicadores de Calidad y construcción de un cuestionario a medida. Aplicación al ambito de asignaturas de Ingeniería Telematica [Evaluation of the quality of Virtual Courses: Quality Indicators and construction of a customized questionnaire. Application to the field of Telematics Engineering subjects] (Doctoral dissertation). Universidad de Extremadura, Spain. Bae, S.-Y., Moon, J., & Morrison, J. R. (2017). Design of engineering courses as a service: Emotions, senses and implementation. International Journal of Engineering Education, 33(5), 1561-1574. Baldomero Ramírez-Fernandez, M. (2015a). MOOC appraisal: A quality perspective. Revista Iberoamericana de Educación a Distancia, 18(2), 171-195. Baldomero Ramírez-Fernandez, M. (2015b). Proposal quality certification of educational Spanish MOOC courses offer conducted by the National Institute of Educational Technologies and Teacher Training. International Journal of Educational Research and Innovation, 3, 121-133. Baldomero Ramírez-Fernandez, M., & Salmerón Silvera, J. L. (2015). Edutool®: a tool for evaluating and accrediting the quality of Moocs [EduTool®: Un instrumento para la evaluación y acreditación de la calidad de los MOOCs]. Educación XX1, 18(2), 97-123. Baldomero Ramírez-Fernandez, M., Salmerón Silvera, J. L., & López Meneses, J. L. (2015). Comparative between quality assessment tools for MOOCs: ADECUR vs Standard UNE 66181: 2012 [Comparativa entre instrumentos de evaluación de calidad de cursos MOOC: ADECUR vs Normas UNE 66181:2012]. RUSC. Universities and Knowledge Society Journal, 12(1), 131-144. Baturay, M. H. (2015). An Overview of the World of MOOCs. Procedia - Social and Behavioral Sciences, 174(Supplement C), 427-433. Bozkurt, A., Akgün-ozbek, E., & Zawacki-Richter, O. (2017). Trends and patterns in massive open online courses: Review and content analysis of research on MOOCs (2008-2015). International Review of Research in Open and Distance Learning, 18(5), 118-147. https://doi.org/10.19173/irrodl.v18i5.3080 Bozkurt, A., Keskin, N. O., & Waard, I. de. (2016). Research Trends in Massive Open Online Course (MOOC) Theses and Dissertations: Surfing the Tsunami Wave. Open Praxis, 8(3), 203-221. https://doi.org/10.5944/openpraxis.8.3.287 Bonafini, F. C., Chae, C., Park, E., & Jablokow, K. W. (2017). How much does student engagement with videos and forums in a MOOC affect their achievement? Online Learning Journal, 21(4), 223-240. https://doi.org/10.24059/olj.v21i4.1270 Cabero Almenara, J., & López Meneses, E. (2009). Descripción de un instrumento didactico para el analisis de modelos y estrategias de enseñanza de cursos universitarios en la red (A.D.E.C.U.R.). [Description of a didactic instrument for the analysis of models and strategies for teaching university courses in the network (A.D.E.C.U.R.)]. Píxel-Bit. Revista de Medios y Educación, 34, 13-30. Cabero-Almenara, J., Marín-Díaz, V., & Sampedro-Requena, B. E. (2017). Research contributions on the educational use of MOOCs. Revista Espanola de Pedagogia, 75(266), 7-27. https://doi.org/10.22550/REP75-1-2017-01 Campbell, J., Gibbs, A., Najafi, H., & Severinski, C. (2014). A comparison of learner intent and behaviour in live and archived MOOCs. International Review of Research in Open and Distance Learning, 15(5), 235-262. Castaño, C., Maíz, I., & Garay, U. (2015). Design, motivation and performance in a cooperative MOOC course. Comunicar, 44, 19-26. Chapman, S. A., Goodman, S., Jawitz, J., & Deacon, A. (2016). A strategy for monitoring and evaluating massive open online courses. Evaluation and Program Planning, 57, 55-63. Chiappe-Laverde, A., Hine, N., & Martínez-Silva, J.-A. (2015). Literature and practice: A critical review of moocs. Comunicar, 22(44), 9-18. https://doi.org/10.3916/C44-2015-01 Chen, J., Xu, W., Xu, H., Peng, W., & Xing, B. (2017). Research on vocational tendency and learning quality. Revista de la Facultad de Ingenieria, 32(6), 257-262. Chiou, Y., & Shih, T. K. (2015). Auto grouping and peer grading system in massive open online course (MOOC). International Journal of Distance Education Technologies, 13(3), 25-43. https://doi.org/10.4018/IJDET.2015070102 Christoforaki, M., & Ipeirotis, P. G. (2015). A system for scalable and reliable technical-skill testing in online labor markets. Computer Networks, 90, 110-120. https://doi.org/10.1016/j.comnet.2015.05.020 Cormier, D., & Siemens, G. (2010). Through the Open Door: Open Courses as Research, Learning, and Engagement. EDUCAUSE Review, 45(4), 31-39. Coursera. (March, 2017). Introducing Our New Infographic: How the World Learns. Retrieved March 24 , 2017 from http://coursera.tumblr.com/post/142363925112/introducing-our-new-infographic-how-the-world Duart, J. M., Roig-Vila, R., Mengual-Andrés, S., & Maseda, D. (2017). The pedagogical quality of MOOCs based on a systematic review of JCR and Scopus publications (2013-2015). Revista Española de Pedagogía, 75(266), 29-46. El Ahrache, S. I., Hassan, B., Tabaa, Y., & Medouri, A. (2013). Massive Open Online Courses: A New Dawn for Higher Education? International Journal on Computer Science and Engineering (IJCSE), 5, 323-327. Eranki, K. L. N., & Moudgalya, K. M. (2016). Comparing the Effectiveness of Self-Learning Java Workshops with Traditional Classrooms. Educational Technology & Society, 19(4), 59-74. Estévez-Ayres, I., Crespo-García, R. M., Fisteus, J. A., & Kloos, C. D. (2013). An algorithm for peer review matching in Massive courses for minimising students’ frustration. Journal of Universal Computer Science, 19(15), 2173-2197. Farhan, M., & Aslam, M. (2017). An Interactive Assessment Framework for Visual Engagement: Statistical Analysis of a TEDx Video. Eurasia Journal of Mathematics, Science and Technology Education, 13(4), 1107-1119. https://doi.org/10.12973/eurasia.2017.00661a Frick, T., & Dagli, C. (2016). MOOCs for Research: The Case of the Indiana University Plagiarism Tutorials and Tests. Technology, Knowledge and Learning, 21(2), 255-276. D. Gamage, S. Fernando and I. Perera, "Quality of MOOCs: A review of literature on effectiveness and quality aspects," 2015 8th International Conference on Ubi-Media Computing (UMEDIA), Colombo, 2015, (pp. 224-229). Ghislandi, P. (2016). “The fun they had” or about the quality of MOOC. Journal of E-Learning and Knowledge Society, 12(3), 99-114. Gisbert, J. y Bonfill, X. (2004). How to perform, evaluate and use systematic reviews and meta-analyzes? [¿Cómo realizar, evaluar y utilizar revisiones sistematicas y metaanalisis?] Gastroenterología y Hepatología, 27(3), 129-149. Goldberg, L. R., Bell, E., King, C., O’Mara, C., McInerney, F., Robinson, A., & Vickers, J. (2015). Relationship between participants’ level of education and engagement in their completion of the Understanding Dementia Massive Open Online Course Approaches to teaching and learning. BMC Medical Education, 15(1). https://doi.org/10.1186/s12909-015-0344-z Gómez Hernandez, P., Monge López, C., & Sebastian Heredero, E. (2016). Quality of a mooc inclusion on education: implementation of various instruments and indicators [Calidad de un MOOC sobre inclusión educativa: Aplicación de varios instrumentos e indicadores]. Revista Ibero-Americana de Estudos em Educação, 11(25). Grant, M. J. y Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26(2), 91-108. Gray, C. (2015). Exploring measures of engagement in open online work based learning: Towards better measures of success in open online learning methods, massive or otherwise. International Journal of Technologies in Learning, 22(3), 41-50. Greene, J. A., Oswald, C. A., & Pomerantz, J. (2015). Predictors of Retention and Achievement in a Massive Open Online Course. American Educational Research Journal, 52(5), 925-955. Hills, L., & Hughes, J. (2016). Assessment worlds colliding? Negotiating between discourses of assessment on an online open course. Open Learning, 31(2), 108-115. https://doi.org/10.1080/02680513.2016.1194747 Hsieh, M.-Y. (2016). Online learning era: Exploring the most decisive determinants of MOOCs in Taiwanese higher education. Eurasia Journal of Mathematics, Science and Technology Education, 12(5), 1163-1188. https://doi.org/10.12973/eurasia.2016.1504a Inventado, P. S., & Scupelli, P. (2017). An online learning collaboratory to address multidisciplinary learning challenges at scale. Interaction Design and Architecture(s), (33), 11-32. Isaksson, R., Garvare, R., Johnson, M., Kuttainen, C., & Pareigis, J. (2015). Sustaining Sweden’s competitive position: lean lifelong learning. Measuring Business Excellence, 19(1), 92-102. Kahan, T., Soffer, T., & Nachmias, R. (2017). Types of participant behavior in a massive open online course. International Review of Research in Open and Distance Learning, 18(6), 1-18. https://doi.org/10.19173/irrodl.v18i6.3087 Khalil, M., Brunner, H., & Ebner, M. (2015). Evaluation Grid for xMOOCs. International Journal of Emerging Technologies in Learning, 10(4), 40-45. Kitchenham, B. A. y Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. Version 2.3 (EBSE-2007-01). Law, P. (2015). Recognising informal elearning with digital badging: evidence for a sustainable business model. Open Praxis, 7(4), 299-310. Leach, M., & Hadi, S. M. (2017). Supporting, categorising and visualising diverse learner behaviour on MOOCs with modular design and micro-learning. Journal of Computing in Higher Education, 29(1), 147-159. https://doi.org/10.1007/s12528-016-9129-6 Leony, D., Muñoz-Merino, P. J., Ruipérez-Valiente, J. A., Pardo, A., & Kloos, C. D. (2015). Detection and evaluation of emotions in massive open online courses. Journal of Universal Computer Science, 21(5), 638-655. Liu, Z., Liu, S., Liu, L., Sun, J., Peng, X., & Wang, T. (2016). Sentiment recognition of online course reviews using multi-swarm optimization-based selected features. Neurocomputing, 185, 11-20. https://doi.org/10.1016/j.neucom.2015.12.036 Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: A systematic study of the published literature 2008-2012. International Review of Research in Open and Distance Learning, 14(3), 202-227. Loftis, E., & Martínez Wormser, J. (2016). Developing online information literacy instruction for the undergraduate art student: a collaborative approach in the context of the framework for information literacy. Art Documentation, 35(2), 241-261. MacKay, J. R. D., Langford, F., & Waran, N. (2016). Massive open online courses as a tool for global animal welfare education. Journal of Veterinary Medical Education, 43(3), 287-301. Masanet, E., Chang, Y., Yao, Y., Briam, R., & Huang, R. (2014). Reflections on a massive open online life cycle assessment course. The International Journal of Life Cycle Assessment, 19(12), 1901-1907. Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of Massive Open Online Courses (MOOCs). Computers & Education, 80, 77-83. https://doi.org/10.1016/j.compedu.2014.08.005 Meek, S. E. M., Blakemore, L., & Marks, L. (2017). Is peer review an appropriate form of assessment in a MOOC? Student participation and performance in formative peer review. Assessment and Evaluation in Higher Education, 42(6), 1000-1013. https://doi.org/10.1080/02602938.2016.1221052 Mengual-Andrés, S., Lloret Catala, C., & Roig Vila, R. (2015). Validation of the Questionnaire of Quality Assessment of Online Courses adapted to MOOC [Validación del Cuestionario de evaluación de la calidad de cursos virtuales adaptado a MOOC]. Ried-Revista Iberoamericana De Educación a Distancia, 18(2), 145-169. Montero, I., & León, O. G. (2007). A guide for naming research studies in Psychology. International Journal of Clinical and Health Psychology, 7(3), 847-862. Moskal, P., Thompson, K., & Futch, L. (2015). Enrollment, engagement, and satisfaction in the BlendKit faculty development open, online course. Journal of Asynchronous Learning Network, 19(4). Muñoz-Merino, P. J., Ruipérez-Valiente, J. A., Delgado, K., Auger, M. A., Briz, S., de, C., & Santalla, S. N. (2017). Flipping the classroom to improve learning with MOOCs technology. Computer Applications in Engineering Education, 25(1), 15-25. https://doi.org/10.1002/cae.21774 Patru, M., & Balaji, V. (Eds.). (2016). Making sense of MOOCs, A Guide for Policy-Makers in Developing Countries. París: Unesco: Commonwealth of Learning. Retrieved from http://unesdoc.unesco.org/images/0024/002451/245122E.pdf Petersen, K., Feldt, R., Mujtaba, S. & Mattsson, M. (2008). Systematic Mapping Studies in Software Engineering. EASE, 8, 68-77. Roig-Vila, R., Mengual Andrés, S., & Suarez Guerrero, C. (2014). Assessment the pedagogical quality of the MOOC [Evaluación de la calidad pedagógica de los MOOC]. El Profesorado, 18(1), 27-41. Ruipérez-Valiente, J. A., Muñoz-Merino, P. J., Pijeira, D., Ruiz, J. S., & Kloos, C. D. (2017). Evaluation of a learning analytics application for open edX platform. Computer Science and Information Systems, 14(1), 51-73. https://doi.org/10.2298/CSIS160331043R Ruiz Bras, I. I. (2016). Mooc in numbers, an analysis to begin reflection [Los MOOC en números, un analisis para comenzar la reflexión]. Revista Digital Universitaria, 17(1), 1-16. Russell, E. J. (2012). The role of servant leadership in faculty development programs: A review of the literature. Turkish Online Journal of Distance Education, 13(1), 15-19. Sa’Don, N. F., Alias, R. A., & Ohshima, N. (2015). Nascent research trends in MOOCs in higher educational institutions: A systematic literature review. Presentado en 2014 International Conference on Web and Open Access to Learning, ICWOAL 2014. https://doi.org/10.1109/ICWOAL.2014.7009215 Sinclair, J., & Kalvala, S. (2016). Student engagement in massive open online courses. International Journal of Learning Technology, 11(3), 218-237. https://doi.org/10.1504/IJLT.2016.079035 Sinclair, J., Boyatt, R., Foss, J., & Rocks, C. (2016). A study of user participation across different delivery modes of a massive open online course. International Journal of Learning Technology, 11(2), 93-113. Smith, M. A. (2015). Output from statistical predictive models as input to elearning dashboards. Future Internet, 7(2), 170-183. https://doi.org/10.3390/fi7020170 Tiejun, Z. (2016). Empirical research on the effectiveness of MOOCs in developing design students on sino-foreign cooperation university programs. International Journal of Emerging Technologies in Learning, 11(6), 19-27. Torres-Coronas, T., & Vidal-Blasco, M.-A. (2017). MOOC and blended learning models: Analysis from a stakeholders’ perspective. International Journal of Information and Communication Technology Education, 13(3), 88-99. https://doi.org/10.4018/IJICTE.2017070107 UNESCO (March, 2017). La UNESCO lanza una guía de cursos por Internet de libre acceso para los países en desarrollo [UNESCO launches a free online course guide for developing countries]. Retrieved 24 March from http://www.unesco.org/new/es/media-services/single-view/news/unesco_launch_for_massive_open_online_courses_guide_for_deve/ Veletsianos, G., & Shepherdson, P. (2016). A systematic analysis and synthesis of the empirical MOOC literature published in 2013-2015. International Review of Research in Open and Distance Learning, 17(2), 198-221. Wittek, L., & Kvernbekk, T. (2011). On the Problems of Asking for a Definition of Quality in Education. Scandinavian Journal of Educational Research, 55(6), 671-684. Yepes-Baldó, M., Romeo, M., Martín, C., García, M. A., Monzó, G., & Besolí, A. (2016). Quality indicators: developing “MOOCs” in the European Higher Education Area. Educational Media International, 53(3), 184-197. Yousef, A. M. F., Chatti, M. A., Schroeder, U., Wosnitza, M., & Jakobs, H. (2014). MOOCs a review of the state-of-the-art (Vol. 3, pp. 9-20). Presentado en CSEDU 2014 - Proceedings of the 6th International Conference on Computer Supported Education. Yousef, A., Chatti, M., Schroeder, U., & Wosnitza, M. (2015). A usability evaluation of a blended MOOC environment: An experimental case study. International Review of Research in Open and Distance Learning, 16(2), 69-93. Zawacki-Richter, O., Bozkurt, A., Alturki, U., & Aldraiweesh, A. (2018). What research says about MOOCs - An explorative content analysis. International Review of Research in Open and Distance Learning, 19(1), 242-259. https://doi.org/10.19173/irrodl.v19i1.3356 Zhu, M., Sari, A., & Lee, M. M. (2018). A systematic review of research methods and topics of the empirical MOOC literature (2014–2016). Internet and Higher Education, 37, 31-39. https://doi.org/10.1016/j.iheduc.2018.01.002