Development of the learning management systems evaluation scale based on transactional distance theory

Abstract

This study aimed to develop the Learning Management Systems Evaluation Scale (LMSES) with reference to Transactional Distance Theory (TDT). At the first stage of the scale development, it was observed that 19 items with three factors explained 63.73% of the total variance. The variance amounts explained by the scale factors are Dialogue (D=8 items, 23.06%), Structure (S=7 items, 25.74%), and Autonomy (A=4 items, 14.93%). As a result of the confirmatory factor analysis performed at the second stage of the scale development, it was determined that the scale had a sufficient reliability coefficient (Cronbach's alpha S=0.90, D=0.89, A=0.82) and fit indices (χ2=252.78; sd=146; χ2/sd =1.73; CFI=0.95, NFI=0.90, GFI=0.89, AGFI=0.85, SRMR=0.06, RMSEA=0.06; p<0.001). The overall Cronbach's alpha coefficient of the scale was calculated as 0.94. Furthermore, the structural reliability of the scale (Sω=0.90, SAVE=0.57; Dω=0.88, DAVE=0.49; Aω=0.82, AAVE=0.54) and the results of the HTMT (rS-D=0.68, rA-D=0.80, rA-S=0.76) correlation ratio analysis were examined, and the evidence for the validity of the scale was supported. According to these research findings, the LMSES has the potential to provide a statistically verified measurement structure based on TDT and support advanced theoretical and statistical studies.

Keywords

• Learning Management Systems
• Transactional Distance Theory
• Scale development
• Structural reliability
• HTMT

References

1. Author (YEAR). Hidden for blind-review process.
2. Akbulut, Y. (2010). Using SPSS in social sciences: Frequently used statistical analyses and solved problems. İdeal Kültür Yayıncılık.
3. Aldrich, J. (1997). R.A. Fisher and the making of maximum likelihood 1912-1922. Statistical Science, 12(3), 162-176. https://doi.org/10.1214/ss/1030037906
4. Arbaugh, J. B., Cleveland-Innes, M., Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., & Swan, K. P. (2008). Developing a community of inquiry instrument: Testing a measure of the community of inquiry framework using a multi-institutional sample. The Internet and Higher Education, 11(3-4), 133-136.
5. Ateş, A. (2013). Eğitsel web sitelerini değerlendirmeye yönelik bir ölçek önerisi [A scale proposal for evaluation of the educational web sites]. Eğitim Teknolojileri Araştırmaları Dergisi, 4(1). Retrieved from http://www.et-ad.net/.
6. Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, The Guilford Press.
7. Browne, M. W., & Cudeck, R. (1993). Alternative ways of assess-ing model fit. In: K. Bollen & J. S. Long, eds. Testing structural equation models, 136–162. Newbury Park, CA: Sage.
8. Bryne, B. M. (2000). Structural equation modeling with Amos: Basic concepts, applications, and programming. Mahwah, NJ:Erlbaum.

9. Büyüköztürk, Ş. (2012). Sosyal bilimler için veri analizi el kitabı (16. Baskı). Ankara: Pegem Akademi Yayıncılık.
10. Byrd, R. (2018). Using appropriate E-learning systems to optimize teaching and learning. GSTF Journal on Computing, 2(3).
11. Byrne, B. M. (1998). Structural equation modeling with LISREL, PRELIS and SIMPLIS: Basic concepts, applications, and programmings. London, Lawrence Erlbaum Assocatiates, Publishers.
12. Çakmak, E. K., Güneş, E., Çiftçi, S., & Üstündağ, M. T. (2011). Developing a web site usability scale: The validity and reliability analysis & implementation results. Pegem Journal of Education and Instruction, 1(2), 31-40.
13. Cohen, L., Manion, L., & Morrison, K. (2007). Research methods in education (6th ed). London and New York: Routledge.
14. Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk, Ş. (2012). Sosyal bilimler için çok değişkenli istatistik: SPSS ve LISREL uygulamaları (2nd edition). Ankara: Pegem Akademi.
15. Comrey, A. L., & Lee, H. B. (1992). Interpretation and application of factor analytic results. In: A First Course on Factor Analysis (2nd edition). Hillsdale, NJ: Lawrence Erlbaum.
16. Demirkol, D., & Şeneler, Ç. (2018). A Turkish translation of the system usability scale: The SUS-TR. Usak University Journal of Social Sciences, 11(3), 237-253. http://dx.doi.org/10.29217/uujss.495
17. dos Santos, P. M., & Cirillo, M. Â. (2021). Construction of the average variance extracted index for construct validation in structural equation models with adaptive regressions. Communications in Statistics-Simulation and Computation, 1-13.
18. Elfeky, A. I. M., Masadeh, T. S. Y., & Elbyaly, M. Y. H. (2020). Advance organizers in flipped classroom via e-learning management system and the promotion of integrated science process skills. Thinking Skills and Creativity, 35, 100622. https://doi.org/10.1016/j.tsc.2019.100622
19. Field, A. (2005). Discovering statistics using SPSS (2nd ed.). London, England: Sage.
20. Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39-50. https://doi.org/10.2307/3151312.
21. Fraenkel, J.R., Wallen, N. E. & Hyun, H. H. (2011). Validity and reliability, how to design and evaluate research in science education (8th Edition). Mc Graw–Hill Companies, 393-394.
22. Gürses, E. A. (2006). Usability in library WEB sites and design based on usability guidelines. Hacettepe University Institute of Social Sciences, Doctoral of Thesis.
23. Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis (7th ed.). Englewood Cliffs, Prentice Hall.
24. Hamzah, M. L., Rukun, K., Rızal, F., & Purwatı, A. A. (2019). A review of increasing teaching and learning database subjects in computer science. Revista ESPACIOS, 40(26). Retrieved from http://www.revistaespacios.com/a19v40n26/a19v40n26p06.pdf
25. Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the academy of marketing science, 43(1), 115-135. doi: 10.1007/s11747-014-0403-8
26. Hodge, V.J., & Austin, J. (2004). A survey of outlier detection methodologies. Artificial Intelligence Review, 22(2), 85-126. Retrieved from https://link.springer.com/article/10.1023/B:AIRE.0000045502.10941.a9
27. Horzum, M. B. (2010). Uzaktan eğitimde uzaklığın boyutları ve tasarımı: coğrafi uzaklığa karşın transaksiyonel (psikolojik ve iletişimsel) uzaklığın azaltılması [Distance in Distance Education]. The Journal of SAU Education Faculty, 20, 95-118.
28. Horzum, M. B. (2011). Developing transactional distance scale and examining transactional distance perception of blended learning students in terms of different variables. Educational Sciences: Theory & Practice, 11(3), 1571-1587.
29. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118
30. Huang, C., Wang, Y., Wu, T., & Wang, P. (2013). An empirical analysis of the antecedents and performance consequences of using the moodle platform. International Journal of Information and Educational Technology, 3(2), 217-221. https://doi.org/10.7763/IJIET.2013.V3.267
31. Khaleghinejad, A., & Ziaaldini, M. (2015). Relationship between employees' safety climate and safety performance with respect to mediating effect of safety knowledge and safety motivation in sarcheshmeh copper complex. Health and Safety at Work, 5(4), 69-86.
32. McIsaac, M.S., & Gunawardena, C.N. (1996). Distance education. Ed: D.H. Jonassen, Handbook of Research for Educational Communications and Technology: A Project of The Association for Educational Communications and Technology, 403-437. New York: Simon & Schuster Macmillan.
33. Mershad, K., & Wakim, P. (2018). A learning management system enhanced with internet of things applications. Journal of Education and Learning, 7(3), 23. http://doi.org/10.5539/jel.v7n3p23
34. Moore, M. G., (1989). Three types of interaction. The American Journal of Distance Education, 3(2), 1-7.
35. Moore, M.G. (1993). Theory of transactional distance. In D. Keegan (ed.), Theoretical Principles of Distance Education, 22-38. New York: Routledge.
36. Moore, M.G., & Kearsley, I.G. (1996). Distance education: A systems view. Wadsworth Publishing Company.
37. Mtebe, J. S., & Raisamo, R. (2014). A model for assessing learning management system success in higher education in sub‐saharan countries. The Electronic Journal of Information Systems in Developing Countries, 61(1), 1-17.
38. Muhardi, M., Gunawan, S. I., Irawan, Y., & Devis, Y. (2020). Design of web based LMS (learning management system) in SMAN 1 kampar kiri hilir. Journal of Applied Engineering and Technological Science (JAETS), 1(2), 70-76. https://doi.org/10.37385/jaets.v1i2.60
39. Noar, S. M. (2003). The role of structural equation modeling in scale development. Structural Equation Modeling, 10(4), 622-647. https://doi.org/10.1207/S15328007SEM1004_8
40. Özonur, M., Kamışlı, H., Yelken, T. Y., & Tokmak, H. S. (2019). Investigation of distance education students’ opinions about the ENOCTA learning management system. Mehmet Akif Ersoy University Journal of Education Faculty, (50), 283-302.
41. Pallant, J. (2007). SPSS survival manual: A step by step guide to data analysis using SPSS for windows. 3 ed. Sydney: McGraw Hill.
42. Raza, S. A., Qazi, W., Khan, K. A., & Salam, J. (2021). Social Isolation and acceptance of the learning management system (LMS) in the time of COVID-19 Pandemic: An expansion of the UTAUT model. Journal of Educational Computing Research, 59(2), 183-208. https://doi.org/10.1177/0735633120960421
43. Sarıkaya, Y. (2014). Okul deneyimi ve öğretmenlik uygulaması dersleri için geliştirilen web tabanlı bir sistemin kullanışlığının incelenmesi. Fırat University Institute of Education Sciences Master of Thesis.
44. Schweizer, K., Moosbrugger, H., & Schermelleh-Engel, K. (2003). Models for hierarchical structures in differential psychology. Methods of Psychological Research Online, 8(2), 159-180. Retrieved from nternet: http://www.mpr-online.de
45. Sinclair, J., & Aho, A. M. (2018). Experts on super innovators: Understanding staff adoption of learning management systems. Higher Education Research & Development, 37(1), 158–172. https://doi.org/10.1080/07294360.2017.1342609
46. Tabachnick, B., & Fidell, L. S. (2001). Using multivariate statistics (4th ed.). Needham Heights: Allyn & Bacon.
47. Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. American Psychological Association.
48. Turan, O. S., & Canal, M. R. (2011). Usability study of learning management system; example of the Gazi English Language School. Journal of Information Technologies, 4(3), 47-52.
49. Walker, J., & Madden, S. (2008). Factor analysis, path analysis, and structural equation modeling. Statistics in Criminology and Criminal Justice: Analysis and Interpretation (3rd ed.). USA: Jones & Bartlett Publishers, 325-51.
50. Yılmaz, V. (2004). LISREL ile yapısal eşitlik modelleri: Tüketici şikayetlerine uygulanması. Anadolu University Journal of Social Sciences, 4(1), 77-90.
51. Zaharias, P., & Pappas, C. (2016). Quality management of learning management systems: A user experience perspective. Current Issues in Emerging eLearning, 3(1), 5.
52. Zwain, A. A. A. (2019). Technological innovativeness and information quality as neoteric predictors of users’ acceptance of learning management system: An expansion of UTAUT2. Interactive Technology and Smart Education, 16(3), 239-254. https://doi.org/10.1108/ITSE-09-2018-0065