EDUCATIONAL DESIGN AND EVALUATION MODELS OF THE LEARNING EFFECTIVENESS IN E-LEARNING PROCESS: A SYSTEMATIC REVIEW

Year 2023, Volume: 24 Issue: 4, 318 - 347, 01.10.2023

Adamantia Spatıotı , Ioannis Kazanıdıs , Jenny Pange

https://doi.org/10.17718/tojde.1177297

Abstract

Educational Design and Evaluation Models are important factors in e-learning as they provide guidance information for proper strategy organization pursuing both specific learning outcomes and ensuring the main elements of e-learning, such as self-regulation and collaborative learning. The examined educational models of ADDIE, Bloom, and Kirkpatrick are widely known and recognized as models for design and measuring the effectiveness of learning in order to achieve the best possible learning outcomes based on the needs of a specific target group in a specific educational context whether traditional or digital. Specifically, the ADDIE Model is a widely known learning design model used by many educational designers and training programmers to develop education and training programs. The Bloom Taxonomy is a method of building learning goals that follows the process of cognition. The Kirkpatrick Model is a method of evaluating the effectiveness of e-learning and educational programs in general, both in terms of training and business performance of learners. The purpose of this paper was both the investigation of the academic performance, the self-regulated learning and the collaborative learning in relation to the models of ADDIE, Kirkpatrick and Bloom in distance online environments and their effectiveness to the learning process. Meta-analysis was applied for research methodology. After a systematic literature review, we found that only 37 articles were appropriate for meta-analysis. Especially, 23 articles were on the ADDIE model, 9 articles were on the Kirkpatrick model and 5 articles were on the Bloom model. According to the results of this study, we found that all models apply to online process and meet different learning requirements. Regarding the cognitive performance of the trainees, all models supported the effectiveness of distance education. Moreover, the self-regulated learning and the collaborative learning, as factors inextricably linked to the effectiveness of the distance education, were examined in a small number studies in the above models. Finally, all three examined models reinforced students’ positive attitudes and perceptions, even while transferring the acquired knowledge to the workplace.

Keywords

ADDIE model, Bloom Taxonomy, Kirkpatrick model, instructional design, e-learning, distance education.

References

• REFERENCES Articles Abernathy, D. (2019). ADDIE in Action: A Transformational Course Redesign Process. Journal for the Advancement of Educational Research 13(1), 8-19.
• Adnan, N. H., & Ritzhaupt, A. D. (2017). Software Engineering Design Principles Applied to Instructional Design: What can we Learn from our Sister Discipline? TechTrends, 62, 77–94.
• Almelhi, A. M. (2021). Effectiveness of the ADDIE Model within an E-Learning Environment in Developing Creative Writing in EFL Students. English Language Teaching, 14(2), 20-36.
• Aluko, F. R., & Shonubi, O. K. (2014). Going beyond Kirkpatrick's Training Evaluation Model: The role of workplace factors in distance learning transfer. Africa Education Review, 11(4), 638-657.
• Amiti, F. (2020). Synchronous and Asynchronous E-Learning. European Journal of Open Education and E-Learning Studies, 5(2), 60-70.
• Azevedo, R., & Jacobson, M. (2008). Advances in scaffolding learning with hypertext and hypermedia: A summary and critical analysis. Educational Technology Research and Development, 56(1), 93–100.
• Azevedo, R., Moos, D. C., Greene, J. A., Winters, F. I., & Cromley, J. G. (2008). Why is externally-regulated learning more effective than self-regulated learning with hypermedia? Educational Technology Research and Development, 56(1), 45–72.
• Ballera, M., Lukandu, I. A., & Radwan, A. (2014). Improving Learning throughput in E-learning using Interactive - Cognitive Based Assessment. The International Journal of E-Learning and Educational Technologies in the Digital Media (IJEETDM) 1(1), 32-49.
• Barari, N., Zadeh, M. R., Khorasani, A., & Alami, F. (2020). Designing and validating educational standards for E-teaching in virtual learning environments (VLEs), based on revised Bloom’s taxonomy. Interactive Learning Environments, 1–13.
• Blau, I., Weiser, O., & Eshet-Alkalai, Y. (2017). How do medium naturalness and personality traits shape academic achievement and perceived learning? An experimental study of face-to-face and synchronous e-learning. Research in Learning Technology, 25(2), 1-21.
• Chang, N., & Chen, L. (2014). Evaluating the Learning Effectiveness of an Online Information Literacy Class Based on the Kirkpatrick Framework. International Journal of Libraries and Information Studies, 64(3), 211-223.
• Cheng, K. W. (2011). A model for developing industry demand-driven e-learning curricula under ADDIE. World Transactions on Engineering and Technology Education (WTE&TE), 9(1), 18-24.
• Chrysafiadi, K & Virvou, M. (2013). PeRSIVA: An empirical evaluation method of a student model of an intelligent e-learning environment for computer programming. Computers & Education, 68(), 322-333.
• De la Fuente, J., Zapata, L., Martínez-Vicente, J. M., Sander, P., & Putwain, D. (2015). Personal Self-regulation, Self-regulated Learning and Coping Strategies, in University Context with Stress. In: Peña-Ayala A. (eds) Metacognition: Fundaments, Applications, and Trends. Intelligent Systems Reference Library, 76, 223-255. Springer, Cham.
• DeRouin, R. E., Fritzsche, B. A., & Salas, E. (2005), "Learner Control and Workplace E-Learning: Design, Person, and Organizational Issues. In Martocchio, J. J. (Ed.) Research in Personnel and Human Resources Management, 24, 181-214. Emerald Group Publishing Limited, Bingley.
• Dogra, A., & Dutt, S. (2016). Effect of Online Learning in Psychology course on Undergraduate students’ Engagement in Learning. Issues and Ideas in Education, 4(1), 21-28.
• Domun, M., & Bahadur, G. K. (2014). Design and Development of a self-assessment tool and investigating its effectiveness for e-learning. European Journal of Open, Distance and e-Learning, 17(1), 1-25.
• Donelan, H. & Kear, K. (2018). Creating and Collaborating: Students’ and Tutors’ Perceptions of an Online Group Project. International Review of Research in Open and Distributed Learning, 19(2). 36-54.
• Durak, G. & Ataizi, M. (2016). The ABC's of Online Course Design According to Addie Model. Universal Journal of Educational Research, 4(9), 2084 - 2091.
• Edwards, M. E., & Black, E. W. (2012). Contemporary Instructor-Librarian Collaboration: A Case Study of an Online Embedded Librarian Implementation. Journal of Library & Information Services in Distance Learning, 6(3-4), 284–311.
• Eiriemiokhale, A. K., & Idiedo, O. V. (2020). Perceptions and Attitude of Students toward E-Learning in Kwara State University, Malete, Kwara State, Nigeria. Journal of Education and Practice, 11(31), 60-67.
• Evans, A. M., Ellis, G., Norman, S., & Luke, K. (2014). Patient safety education-A description and evaluation of an international, interdisciplinary e-learning programme. Nurse Education Today, 34(2), 248-251.
• Farhat H. N. (2021). Use of Digital Assessments How to Utilize Digital Bloom to Accommodate Online Learning and Assessments? Asian Journal of Education and Training, 7(1), 30-35.
• Fernandes, R. A. M. L., de Oliveira Lima, J. T., da Silva, B. H., Sales, M. J. T., & de Orange, F. A. (2020). Development, implementation and evaluation of a management specialization course in oncology using blended learning.BMC Medical Education, 20(37), 1-10.
• Galanis, P. (2009). Systematic review and meta-analysis. Archives of Greek Medicine, 26 (6): 826-841.
• Galloway, D. L. (2005). Evaluating distance delivery and e-learning is kirkpatrick's model relevant? Performance Improvement, 44(4), 21–27.
• Gambrari, I. A., Yusuf, M. O., & Thomas, D. A. (2015). Effects of Computer-Assisted STAD, LTM and ICI Cooperative Learning Strategies on Nigerian Secondary School Students' Achievement, Gender and Motivation in Physics. Journal of Education and Practice, 6(19), 16-28.
• Goh, W. W., Wong, S. Y., & Ayub, E. (2018). The Effectiveness of MOOC among Learners based on Kirkpatrick’s Model. In: Tang S., Cheah S. (eds) Redesigning Learning for Greater Social Impact, p.p. 313-323. Springer, Singapore.
• Greene, J. A., Seung, B. Y., & Copeland, D. (2014). Measuring Critical Components of Digital Literacy and their Relationships with Learning. Computers & Education, 76.
• Hanafi, Y, Murtadho, N., Ikhsan, M. A., & Diyana, T. N. (2020). Reinforcing Public University Student’s Worship Education by Developing and Implementing MobileLearning Management System in the ADDIE Instructional Design Model. International Journal of Interactive Mobile Technologies (iJIM), 4(2), 215-241.
• Hess, A. N. & Greer, K. (2016). Designing for Engagement: Using the ADDIE Model to Integrate High-Impact Practices into an Online Information Literacy Course. Communications in Information Literacy, 10 (2), 264-282.
• Hsu, T-C, Lee-Hsieh, J., Turton, M. A., & Cheng, S-F (2014). Using the ADDIE Model to Develop Online Continuing Education Courses on Caring for Nurses in Taiwan. The Journal of Continuing Education in Nursing, 45(3), 124–131.
• Hubalovsky, S., Hubalovska, M., Musilek, M. (2018). Assessment of the influence of adaptive E-learning on learning effectiveness of primary school pupils. Computers in Human Behavior, 92(), 691-705.
• Ismail, M., E., Utami, P., Ismail, I., M., Hamzah, N., & Harun, H. (2018). Development of massive open online course (MOOC) based on addie model for catering courses. Journal Pendidikan Vokasi, 8(2), 184-192.
• Johnson, D. W., & Johnson, R. T. (2011). Cooperative Learning.In Christie, D., J., The Encyclopedia of Peace Psychology. Wiley-Blackwell.
• Jusas, V., Butkiene, R., Venˇckauskas, A., Burbaite, R., Gudoniene, D., Grigaliunas, Š. & Andone, D. (2021). Models for Administration to Ensure the Successful Transition to Distance Learning during the Pandemic. Sustainability, 13(4751), 1-22.
• Keengwe, J., Onchwari, G., Agamba, J. (2014). Promoting effective e-learning practices through the constructivist pedagogy. Educ. Inf. Technol., 19, 887–898.
• Kennedy, P. E., Chyung, S. Y., Winiecki, D. J., & Brinkerhoff, R. O. (2014). Training professionals' usage and understanding of Kirkpatrick's level 3 and level 4 evaluations. International Journal of Training and Development, 18(1), 1-21.
• Khalil, M. K., & Elkhider, I. A. (2016). Applying learning theories and instructional design models for effective instruction. Advances in Physiology Education 40, 147–156.
• Khare, A. P., & Kumar, J. (2015). A Framework for Evaluation of E- learning Applications in Developing Countries. Advances in Computer Science and Information Technology (ACSIT) 2(13), 62-67.
• Kisanga, D. H., & Ireson, G. (2016). Test of e-Learning Related Attitudes (TeLRA) scale: Development, reliability and validity study. International Journal of Education and Development using Information and Communication Technology (IJEDICT), 12(1), 20-36.
• Koç, E. (2020). Design and Evaluation of a Higher Education Distance EAP Course by using the ADDIE model. Electronic Journal of Social Sciences, 19(73), 522-531.
• Kuciapski, M. (2010). Model for Project Management for Development and Implementation of E-Learning Courses. In: Forbrig P., Günther H. (eds) Perspectives in Business Informatics Research. BIR 2010. Lecture Notes in Business Information Processing, 64(4), 43-54. Springer, Berlin, Heidelberg.
• Kumpas-Lenk, K., Eisenschmidt, E., & Veispak, A. (2018). Does the design of learning outcomes matter from students’ perspective? Studies in Educational Evaluation, 59(2018), 179-186.
• Lahti, M., Kontio, R., Pitkänen, A., Välimäki, M. (2014). Knowledge transfer from an e-learning course to clinical practice. Nurse Education Today, 34 (5), 842-847.
• Lameras, P., Arnab, S. (2022). Power to the Teachers: An Exploratory Review on Artificial Intelligence in Education. Information, 13, 14.
• Lau, K. H., Lam, T., Kam, B. H., Nkhoma, M., Richardson, J., & Thomas, S. (2017). The role of textbook learning resources in e-learning: A taxonomic study. Computers & Education, 118, March 2018, 10-24.
• Lavender, D. T., Omoni, G., Lee, K., Wakasiaki, S., Campbell, M., Watiti, J., & Mathai, M. (2013). A pilot quasi-experimental study to determine the feasibility of implementing a partograph e-learning tool for student midwife training in Nairobi. Midwifery, 29(8), 876–884.
• Liberati, A, Altman D. G, Tetzlaff J, Mulrow C, Gøtzsche P, ., Ioannidis, J.P.A., Clarke, M., Devereaux, P.J., Kleijnen, J., Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLoS Med 6(7): e1000100.
• Lin, J., & Cantoni, L. (2018). Decision, Implementation, and Confirmation: Experiences of Instructors behind Tourism and Hospitality MOOCs. The International Review of Research in Open and Distributed Learning, 19(1), 275-293.
• López-Zambrano, J., Lara, J. A. & Romero, C. (2022) Improving the portability of predicting students’ performance models by using ontologies. Journal of Computing in Higher Education 34, 1–19.
• Mabrouk, P., A. (2007). Active Learning: Models from the Analytical Sciences. In R., M., Felder, & R., Brent, Cooperative Learning, 970, 34-53. ACS Symposium Series.
• Mavroudi, A., & Hadzilacos, T. (2013). Learning Needs Analysis of Collaborative E-Classes in Semi-Formal Settings: The REVIT Example. The International Review of Research in Open and Distributed Learning (IRRODL) 14(5), 211-239.
• Moher, D., Liberati, A, Tetzlaff, J, Altman D. G., The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097.
• Moreira, I. C., Ramosa, I., Venturac, S.R., & Rodrigues, P.P. (2019). Learner’s perception, knowledge and behaviour assessment within a breast imaging E-Learning course for radiographers. European Journal of Radiology, 111(2019), 47–55.
• Mullins, K. (2014). Good IDEA: instructional design model for integrating information literacy. Journal of Academic Librarianship, 40(3), 339-349.
• Nordin, N., Embi, M. A., Norman, H. (2016). Towards Envisioning the Future of Learning in Malaysia: Development of a Malaysia MOOC Based on the Iterative ADDIE Instructional Design Framework. In: Luaran J., Sardi J., Aziz A., Alias N. (eds) Envisioning the Future of Online Learning, 9(24), 269-279. Springer, Singapore.
• Ozdileka, Z., & Robeck, E. (2009). Operational priorities of instructional designers analyzed within the steps of the Addie instructional design model. Procedia - Social and Behavioral Sciences, 1(1), 2046-2050.
• Author (2014). Deleted for peer review.
• Patel, S. R., Margolies, P. J., Covell, N. H., Lipscomb, C., & Dixon, L. B. (2018). Using Instructional Design, Analyze, Design, Develop, Implement, and Evaluate, to Develop e-Learning Modules to Disseminate Supported Employment for Community Behavioral Health Treatment Programs in New York State.Frontiers in Public Health, 6(113), 1–9.
• Paull, M., Whitsed, C., & Girardi, A. (2016). Applying the Kirkpatrick model: Evaluating an Interaction for Learning Framework curriculum intervention. Issues in Educational Research, 26(3), 490-507.
• Authors (2019). Deleted for peer review.
• Authors (2016). Deleted for peer review.
• Puzziferro, M. Shelton, K. (2008). A Model for Developing High-Quality Online Courses: Integrating a Systems Approach with Learning Theory. Journal of Asynchronous Learning Networks, 12(3-4), 119-136.
• Rabak, L., & Cleveland-Innes, M. (2006). Acceptance and resistance to corporate e-learning: A case from the retail sector. International Journal of E-Learning & Distance Education, 21(2), 115-134.
• Reinbold, S. (2013). Using the ADDIE Model in Designing Library Instruction. Medical Reference Services Quarterly, 32(3), p.p. 244–256.
• Reio, T. G., Rocco, T. S., Smith, D. H., & Chang, E. (2017). A Critique of Kirkpatrick's Evaluation Model. New Horizons in Adult Education and Human Resource Development, 29(2), 35–53.
• Robinson, B. K., Dearmon, V. (2013). Evidence-Based Nursing Education: Effective Use of Instructional Design and Simulated Learning Environments to Enhance Knowledge Transfer in Undergraduate Nursing Students. Journal of Professional Nursing, 29(4), 203–209.
• Salas-Rueda, R. A., Salas-Rueda, E. P., & Salas-Rueda, R. D. (2020). Analysis and design of the web game on descriptive statistics through the ADDIE model, data science and machine learning. International Journal of Education in Mathematics, Science and Technology (IJEMST), 8(3), 245-260.
• Santally, M. I., Rajabalee, Y., & Cooshna-Naik, D. (2012). Learning Design Implementation for Distance e-Learning: Blending Rapid e-Learning Techniques with Activity - Based Pedagogies to Design and Implement a Socio-Constructivist Environment. European Journal of Open, Distance and E-Learning, 1-14.
• Santiari, Ni P. L. (2015). Embed Attitude from Student on ELearning Using Instructional Design with ADDIE Model. International Journal of Advanced Computer Science and Applications (IJACSA), 6(11), 35-43.
• Sharif, A., & Cho, S. (2015). 21st-Century Instructional Designers: Bridging the Perceptual Gaps between Identity, Practice, Impact and Professional Development. International Journal of Educational Technology in Higher Education, 12(3), 72–85.
• Song, L., Singleton, E., Hill, J., R., & Koh, M. H. (2004). Improving Online Learning: Student Perceptions of Useful and Challenging Characteristics. Internet and Higher Education, 7(1), 59-70.
• Soto, V. (2013). Which Instructional Design Models are Educators Using to Design Virtual World Instruction? Journal of Online Learning and Teaching, JOLT (MERLOT), 9(3), 364-375.
• Spatiotis, N., Periko, I., Mporas, I., Paraskevas, M. (2020). Sentiment Analysis of Teachers Using Social Information in Educational Platform Environments. Int. J. Artif. Intell. Tools, 29, 2040004.
• Svirko, E., & Mellanby, J. (2009). Attitudes to e-learning, learning style and achievement in learning neuroanatomy by medical students. Medical Teacher, 30(9-10), e219–e227.
• Authors (2014). Deleted for peer review.
• Troussas, C., Krouska, A., & Sgouropoulou, C. (2022). Enriching Mobile Learning Software with Interactive Activities and Motivational Feedback for Advancing Users’ High-Level Cognitive Skills. Computers, 11(2), 1-12.
• Trust, T., & Pektas, E. (2018). Using the ADDIE Model and Universal Design for Learning Principles to Develop an Open Online Course for Teacher Professional Development. Journal of Digital Learning in Teacher Education, 34(4), 219–233.
• Türker, M. F. (2016). Design process for online websites created for teaching Turkish as a foreign language in web based environments. Educational Research and Reviews, 11(8), 642-655.
• Wang, T. H. (2011). Developing Web-based assessment strategies for facilitating junior high school students to perform self-regulated learning in an e-Learning environment. Computers & Education, 57(2), 1801–1812.
• Wang, S. L., Hong, H. T. (2018). The roles of collective task value and collaborative behaviors in collaborative performance through collaborative creation in CSCL. Educational Technology Research and Development 66, 937–953.
• Weiser, O., Blau, I., Eshet-Alkalai, Y. (2018). How do medium naturalness, teaching-learning interactions and Students' personality traits affect participation in synchronous E -learning? The Internet and Higher Education, 37(), 40-51.
• Yu, S.-J., Hsueh, Y.-L., Sun, J.C.-Y,; Liu, H.-Z. (2021). Developing an intelligent virtual reality interactive system based on the ADDIE model for learning pour-over coffee brewing. Comput. Educ. Artif. Intell., 2, 100030.
• Theses and Dissertations Battle, E. L. (2019). Agile Learning versus ADDIE: The Choice for Instructional Designers in Online Learning Development in Higher Education (Doctoral dissertation). Northcentral University. Available from ProQuest Dissertations and Theses database. (UMI No. 13899639).
• Diamantopoulou, K. (2017). Comparative study of educational design models for e-learning. Advantages, disadvantages, applications and theoretical background (Master's dissertation). University of the Aegean, Rhodes. Available from HELLANICUS, Institutional Repository of the Library of the Aegean University, Dissertation I.D. 18140.
• Gournakis, K. (2020). Development and evaluation of an e-learning system for Software Quality (Master's dissertation). Hellenic Open University, Dissertation I.D. 49126
• Hatziroufa, B. (2019). Design, implementation, distance evaluation of an in-company staff training program of Hellenic Open University (Master's dissertation). Hellenic Open University, Dissertation I.D. 43801.
• Manitsara, A. (2020). Design, Development, Implementation and Evaluation of the E-Learning Educational Program: Learning to Use and Manage an Electronic Patient File in the SAP Information System, for the Specialized Doctors of Papageorgiou General Hospital (Master's dissertation). Hellenic Open University, Dissertation I.D. 48813.
• Zampelis, K. (2020). Educational design of Massive Open Online Courses (MOOC) (Master's dissertation). National kapodistrian University of Athens, Dissertation I.D. 2896987. Books Branch, R. & Merrill, M. D. (2011). Characteristics of instructional design models. In R. A. Reiser & J.V. Dempsey (Eds.), Trends and issues in instructional design and technology (pp. 8–16). Upper Saddle River, NJ: Merrill-Prentice Hall.
• Cohen, L., & Manion, L. (1994). Methodology of Educational Research. Athens: Metaixmio.
• Creswell, J. (2016). Research in Education: Design, Conduct and Evaluation of Quantitative and Qualitative Research (trans: KoubarakouN. 2nd ed.).Athens: Ion (year of publication of the original 2015).
• Trivella, L. (2017) Information Communication Technology (ICT) in Open and Distance Learning (ODL): A Tool with Potentials in the Field of Education. The Case Study of Greek Educational Institutions. In: A. Kavoura, D. Sakas, P. Tomaras (Eds), Strategic Innovative Marketing. Springer Proceedings in Business and Economics (pp. 601-608). Switzerland: Springer.
• Tsiolis, G. (2017). Thematic Analysis of Quality Data. Additional educational material in Thematic Unit ΕΚΠ51 –Postgraduate Program: Sciences of Education. Hellenic Open University.
• Zemelman, S., Daniels, H., & Hyde, A. A. (2005). Best practice: today's standards for teaching and learning in America's schools (3rd ed.). Portsmouth, N.H.: Heinemann.
• Internet Resources Research Initiatives. (n.d). University of Cyprus, Library. Retrieved May 05, 2021 from http://library.ucy.ac.cy/el/services/library-guides/evaluation_intresources
• Conference Papers Alturkistani, A., Car, J., Majeed, A., Brindley, D.,Wells, G., & Meinert, E. (2018). Determining the effectiveness of a massive Open online course in data science for health. International Association for Development of the Information Society (IADIS) International Conference on e-Learning, 2018, (pp. 27-36). Madrid, Spain: IADIS.
• Ballera, M., & Elssaedi, M. M. (2013). New e-learning strategy paradigm: a multi-discplinary approach to enhance learning delivery. Proceedings of the 2nd E-Learning Regional Conference - State of Kuwait, 2013. State of Kuwait.
• Gowda, R. S., & Suma, V. (2017). A comparative analysis of traditional education system vs. e-Learning. IEEE, International Conference on Innovative Mechanisms for Industry Applications (ICIMIA), 2017, (pp. 567–571).
• Huang, C.H , Chiu, C.F., Chin, S.L, Hsin, L.H., Yu, Y.P. (2010). A Sports E-learning Platform: Teaching and learning by using Multimedia contents. 3rd IEEE International Conference on Ubi-Media Computing, 2010, (pp. 222–226).