Development of a Scale to Evaluate Virtual Learning Environment Satisfaction

Öz

Recent advances in information and communication technologies (ICT) have resulted in improvements in the delivery of education. It is a well-known fact that learning technologies currently have a pivotal role in education. Amongst them, Virtual Learning Environments (VLEs) are widely used in education. The role of VLEs in improving quality and interaction in education as well as enabling better achievement through the use of a wealth of activities in teaching and learning is widely reported in the literature. However, there is a gap regarding the development of measurement instruments, especially in the Turkish context. Therefore, this study reports the development of a scale to evaluate students’ satisfaction with respect to the use of VLEs in educational settings to address this gap. The dimensions of the scale are contribution (CONT), satisfaction (SAT), and communication (COM), and the scale is formed of 13 items. The sample consists of students enrolled in the Department of Computer Education and Instructional Technologies, studying on blended and face-to-face learning programs. First, the reliability of the instrument was calculated by Cronbach Alpha coefficient and test-retest reliability correlation coefficient. The Cronbach Alpha coefficients were found to be 0.87, 0.83, and 0.81 for CONT, SAT, and COM sub-dimensions respectively. The overall reliability of the scale was 0.92. EFA and CFA were conducted on the data collected from two different sample groups (206 and 186 students for EFA and CFA respectively) for the validity analyses of the scale. Results confirm that the scale is valid and reliable. While the t-test analysis shows no significant difference between gender groups, ANOVA revealed significant differences when year of study is considered.

Anahtar Kelimeler

• irtual learning environment
• V
• Scale
• Reliability
• Validity

Development of a Scale to Evaluate Virtual Learning Environment Satisfaction

Abstract

Recent advances in information and communication technologies (ICT) have resulted in improvements in the delivery of education. It is a well-known fact that learning technologies currently have a pivotal role in education. Amongst them, Virtual Learning Environments (VLEs) are widely used in education. The role of VLEs in improving quality and interaction in education as well as enabling better achievement through the use of a wealth of activities in teaching and learning is widely reported in the literature. However, there is a gap regarding the development of measurement instruments, especially in the Turkish context. Therefore, this study reports the development of a scale to evaluate students’ satisfaction with respect to the use of VLEs in educational settings to address this gap. The dimensions of the scale are contribution (CONT), satisfaction (SAT), and communication (COM), and the scale is formed of 13 items. The sample consists of students enrolled in the Department of Computer Education and Instructional Technologies, studying on blended and face-to-face learning programs. First, the reliability of the instrument was calculated by Cronbach Alpha coefficient and test-retest reliability correlation coefficient. The Cronbach Alpha coefficients were found to be 0.87, 0.83, and 0.81 for CONT, SAT, and COM sub-dimensions respectively. The overall reliability of the scale was 0.92. EFA and CFA were conducted on the data collected from two different sample groups (206 and 186 students for EFA and CFA respectively) for the validity analyses of the scale. Results confirm that the scale is valid and reliable. While the t-test analysis shows no significant difference between gender groups, ANOVA revealed significant differences when year of study is considered.

Keywords

• Virtual learning environment
• Scale
• Reliability
• Validity

References

1. Al-Khalifa, H. S. (2009). JUSUR: The Saudi Learning Management System. In Proceedings of 2nd Annual Forum on e-Learning Excellence in the Middle East, Dubai, UAE.
2. Asoodar, M., Vaezi, S., & Izanloo, B. (2016). Framework to improve e-learner satisfaction and further strengthen e-learning implementation, Computers in Human Behavior, 63, 704-716.
3. Bagozzi, R. P., & Yi, Y. (1988). On the evaluation of structural equation models. Journal of the Academy of Marketing Science, 16(1), 74-94.
4. Barker, J., & Gossman, P. (2013). The learning impact of a virtual learning environment: students’ views teacher education advancement network. Journal University of Cumbria, 5(2), 19-38.
5. Bartlett, M. S. (1954). A note on the multiplying factors for various χ 2 approximations. Journal of the Royal Statistical Society. Series B (Methodological), 16(2), 296-298.
6. Bell, M., & Farrier, S. (2008). Measuring success in e-learning – a multi-dimensional approach. The Electronic Journal of e- Learning, 6(2), 99-110.
7. Beluce, A. C., & Oliveira, K. L. D. (2015). Students’ motivation for learning in virtual learning environments. Paidéia (Ribeirão Preto), 25(60), 105-113.
8. Bentler, P.M. (1980). Multivariate analysis with latent variables: Causal modeling. Annual Review of Psychology, 31, 419-456.

9. Bentler, P.M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588-606.
10. Bermingham V. (2016). Student feedback. In C. Ashford & J. Guth (Eds.), The legal academic’s handbook (pp. 99-101). London, the UK: Palgrave Macmillan.
11. Briggs, R. O., Reinig, B. A., & de Vreede, G. J. (2008). The yield shift theory of satisfaction and its application to the IS/IT domain. Journal of the Association for Information Systems, 9(5), 267-293.
12. Briggs, R. O., Reinig, B. A., & de Vreede, G. J. (2014). An empirical field study of the Yield Shift Theory of satisfaction. 47th Hawaii International Conference on In System Sciences (HICSS), 492-499.
13. Brown, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Beverly Hills, CA: Sage.
14. Bryman, A., & Cramer, D. (1999). Quantitative data analysis with SPSS release 8 for Windows. A guide for social scientists. London and New York: Taylor & Francis Group.
15. Büyüköztürk, Ş. (2012). Sosyal bilimler için veri analizi el kitabı. Ankara: Pegem Yayıncılık.
16. Cokluk, O., Sekercioglu, G., & Buyukozturk, S. (2012). Sosyal bilimler icin cok degiskenli SPSS ve LISREL uygulamalari. Ankara: Pegem Yayincilik.
17. Byrne, B.M., & Campbell, T.L. (1999). Cross-cultural comparisons and the presumption of equivalent measurement and theoretical structure: A look beneath the surface. Journal of Cross-Cultural Psychology, 30, 555-574.
18. Cassidy, S. (2016). Virtual learning environments as mediating factors in student satisfaction with teaching and learning in higher education. Journal of Curriculum and Teaching, 5(1), 113-123.
19. Cheng, K. W. (2011). The gap between e-learning managers and users on satisfaction of e-learning in the accounting industry. Journal of Behavioral Studies in Business, 3, 70-79.
20. Cheng, X., Wang, X., Huang, J. & Zarifis, A. (2016). An experimental study of satisfaction response: evaluation of online collaborative learning. International Review of Research in Open and Distributed Learning, 17(1), 60-78.
21. Ching, Y. H., & Hsu, Y. C. (2015). Online Graduate Students' Preferences of Discussion Modality: Does Gender Matter? Journal of Online Learning and Teaching, 11(1), 31.
22. Chou, S. W., & Liu, C. H. (2005). Learning effectiveness in a Web‐based virtual learning environment: A learner control perspective. Journal of Computer Assisted Learning, 21(1), 65-76.
23. Chua, C., & Montalbo, J. (2014). Assessing students’ satisfaction on the use of Virtual Learning Environment (VLE): An input to a campus-wide e-learning design and implementation. Information and Knowledge Management, 3(4), 108-116.
24. Cutmore, T. R., Hine, T. J., Maberly, K. J., Langford, N. M., & Hawgood, G. (2000). Cognitive and gender factors influencing navigation in a virtual environment. International Journal of Human-Computer Studies, 53(2), 223-249.
25. De Lange, P., Suwardy, T., & Mavondo, F. (2003). Integrating a virtual learning environment into an introductory accounting course: determinants of student motivation. Accounting Education, 12(1), 1-14.
26. Eom, S. B., Wen, H. J., & Ashill, N. (2006). The determinants of students' perceived learning outcomes and satisfaction in university online education: An empirical investigation. Decision Sciences Journal of Innovative Education, 4(2), 215-235.
27. Field, A. (2009). Discovering statistics Using SPSS. London: SAGE Publications Ltd.
28. Fornell, C., & Larcker, D. F. (1981). Structural equation models with unobservable variables and measurement error: Algebra and statistics. Journal of Marketing Research, 382-388.
29. Forteza, F. R., Oltra, A., Miquel, A., & Coy, R. P. (2015). University students and virtual learning environments: motivation, effectiveness and satisfaction. Social & Economic Revue, 13(4), 50-54.
30. Goulão, M. D. F. (2013). Virtual learning styles: does gender matter?. Procedia-Social and Behavioral Sciences, 106, 3345-3354.
31. Gulbahar, Y. (2012). Study of developing scales for assessment of the levels of readiness and satisfaction of participants in e-learning environments. Ankara University Journal of Faculty of Educational Sciences, 45(2), 119-137.
32. Gunn, C., McSporran, M., Macleod, H., & French, S. (2003). Dominant or different? Gender issues in computer supported learning. Journal of Asynchronous Learning Networks, 7(1), 14-30.
33. Jöreskog, K.G., & Sörbom, D. (1993). LISREL 8: User’s guide. Chicago: Scientific Software.
34. Jöreskog, K. G., & Sörbom, D. (1996). LISREL 8: User's reference guide. Scientific Software International.
35. Hair, J. F., Black, B., Babin, B., Anderson, R. E., & Tahtam, R. L. (2006). Multivariate data analysis. Upper Saddle River: Prentice Hall.
36. Hettiarachchi, S., & Wickramasinghe, S. (2016). Impact of virtual learning for improving quality of learning in higher education. 2 nd International Conference on Education and Distance Learning – 1 st July 2016, Colombo, Sri Lanka.
37. Hew, T. S., & Syed Abdul Kadir, S. L. (2016). Predicting instructional effectiveness of cloud-based virtual learning environment. Industrial Management & Data Systems, 116(8), 1557-1584.
38. Horvat, A., Dobrota, M., Krsmanovic, M., & Cudanov, M. (2015). Student perception of Moodle learning management system: a satisfaction and significance analysis. Interactive Learning Environments, 23(4), 515-527.
39. Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39(1), 31-36.
40. Kember, D., & Ginns, P. (2012). Evaluating teaching and learning: A practical handbook for colleges, universities and the scholarship of teaching. New York, NY: Routledge.
41. Kline, R.B. (2011). Principles and practice of structural equation modeling. New York, NY: The Guilford Press.
42. Kolburan-Gecer, A., & Deveci-Topal, A. (2015). Development of satisfaction scale for e-course: Reliability and validity study. Journal of Theory and Practice in Education, 11(4), 1272-1287.
43. Koskela, M., Kiltti, P., Vilpola, I., & Tervonen, J. (2005). Suitability of a Virtual Learning Environment for Higher Education. Electronic Journal of e-Learning, 3(1), 23-32.
44. Ku, H. Y., Tseng, H. W., & Akarasriworn, C. (2013). Collaboration factors, teamwork satisfaction, and student attitudes toward online collaborative learning. Computers in Human Behavior, 29(3), 922-929.
45. Lang, B. A., Dolmans, D.H.J.M., Muijtjens, A.M.M., & van der Vieuten, C.P.N. (2006). Student perceptions of a virtual learning environment for a problem-based learning undergraduate medical curriculum. Medical Education, 40(6), 568-575. http://dx.doi.org/10.1111/j.1365-2929.2006.02484.x
46. Lee, J., Hong, N. L., & Ling, N. L. (2001). An analysis of students' preparation for the virtual learning environment. The Internet and Higher Education, 4(3), 231-242.
47. Lee, S. J., Srinivasan, S., Trail, T., Lewis, D., & Lopez, S. (2011). Examining the relationship among student perception of support, course satisfaction, and learning outcomes in online learning. The Internet and Higher Education, 14(3), 158-163.
48. Leese, M. (2009). Out of class—out of mind? The use of a virtual learning environment to encourage student engagement in out of class activities. British Journal of Educational Technology, 40(1), 70-77.
49. Liaw, S. (2008). Investigating students’ perceived satisfaction, behavioural intention, and effectiveness of e-learning: A case study of the Blackboard system. Computers and Education, 51(2), 864–873. http://dx.doi.org/10.1016/j.compedu.2007.09.005
50. Lin, W. S. (2012). Perceived fit and satisfaction on web learning performance: IS continuance intention and task-technology fit perspectives. International Journal of Human-Computer Studies,70(7), 498–507. Special Issue on User Experience (UX) in Virtual Learning Environments, http://dx.doi.org/10.1016/j.ijhcs.2012.01.006
51. Loewen, S., & Gonulal, T. (2015). Exploratory factor analysis and principal component analysis. In L. Plonsky (Ed.), Advancing quantitative methods in second language research (pp. 182-212). New York, NY: Routledge.
52. Lowes, S., Lin, P., & Kinghorn, B. R. (2016). Gender differences in online high school courses. Online Learning, 20(4), 100-117.
53. Love, N., & Fry, N. (2006). Accounting students’ perceptions of a virtual learning environments: Springboard or safety net? Accounting Education, 15(2), 151-166. http://dx.doi.org/10.1080/06939280600609201
54. MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130-149.
55. Marsh, H.W., Hau, K.T., Artelt, C., Baumert, J., & Peschar, J.L. (2006). OECD’s brief self-report measure of educational psychology’s most useful affective constructs: Cross-cultural, psychometric comparisons across 25 countries. International Journal of Testing, 6(4), 311-360.
56. Martins, L. L., & Kellermanns, F. W. (2004). A model of business school students' acceptance of a web-based course management system. Academy of Management Learning & Education, 3(1), 7-26.
57. McGill, T. J., & Hobbs, V. J. (2008). How students and instructors using a virtual learning environment perceive the fit between technology and task. Journal of Computer Assisted Learning, 24(3), 191-202.
58. Naveh, G., Tubin, D., & Pliskin, N. (2010). Student LMS use and satisfaction in academic institutions: The organizational perspective. Internet and Higher Education, 13, 127–133.
59. OECD (2005). E-learning in Tertiary Education: Where do we stand? Paris: Centre for Educational Research and Innovation.
60. Ogba, I. E., Saul, N., & Coates, N. F. (2012). Predicting students’ attitudes towards advertising on a university Virtual Learning Environment (VLE). Active Learning in Higher Education, 13(1), 63–75.
61. Pallant, J. (2005). SPSS survival manual: A step by step guide to data analysis using SPSS for windows. Australia: Australian Copyright.Perkowski, J. (2013). The role of gender in distance learning: A meta-analytic review of gender differences in academic performance and self-efficacy in distance learning. Journal of Educational Technology Systems, 41(3), 267-278.
62. Pituch, K. A., & Lee, Y.-K. (2006). The influence of system characteristics on e-learning use. Computers & Education, 47, 222–244.
63. Reinig, B. A., Briggs, R. O., & Vreede, G. J. de. (2009). A cross-cultural study of the relationship between perceived changes in likelihood of goal attainment and satisfaction with technologically supported collaboration. International Journal fore-Collaboration, 5(2), 61-74.
64. Rienties, B., Giesbers, B., Lygo-Baker, S., Ma, H. W. S. & Rees, R. (2016). Why some teachers easily learn to use a new virtual learning environment: A technology acceptance perspective. Interactive Learning Environments, 24(3), 539-552. DOI:10.1080/10494820.2014.881394.
65. Schermelleh-Engel, K., & Moosbrugger, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23-74.
66. Shubina, M. (2016). Usability evaluation of MOOC platforms. Usability evaluation of massive open online course platforms. Bachelor’s Thesis Degree Programme in Business Information Technology, Haaga Helia University of Applied Sciences.
67. Stricker, D., Weibel, D., & Wissmath, B. (2011). Efficient learning using a virtual learning environment in a university class. Computers & Education, 56(2), 495-504.
68. Šumak, B., Polancic, G., & Hericko, M. (2010, February). An empirical study of virtual learning environment adoption using UTAUT. In Mobile, Hybrid, and On-Line Learning, 2010. ELML'10. Second International Conference on (pp. 17-22). IEEE.
69. Šumak, B., Hericko, M., Pusnik, M., & Polancic, G. (2011). Factors affecting acceptance and use of Moodle: An empirical study based on TAM, Informatica, 35, 91–100.
70. Sun, P., Tsai, J. R., Finger, G., Chen, Y., & Yeh, D. (2008). What drives a successful e- Learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers & Education, 50(4), 1183−1202.
71. Tabachnick, B.G., & Fidell, L.S. (2007). Using multivariate statistics (5th ed.). Boston: Allyn and Bacon.
72. UCISA Report (2014). 2014 Survey of technology enhanced learning for higher education in the UK. Retrieved on October 17, 2016 from https://www.ucisa.ac.uk/publications/tel_casestudies2016.
73. Unsal, H. (2012). Harmanlanmış öğrenmenin başarı ve motivasyona etkisi. Journal of Turkish Educational Sciences, 10(1), 1-27.
74. Usta, I., Uysal, O., & Okur, M. R. (2016). Çevrimiçi öğrenme tutum olcegi: Gelistirilmesi, gecerligi ve guvenirligi. Journal of International Social Research, 9(43), 2215-2222.
75. Van Raaij, E. M., & Schepers, J. J. (2008). The acceptance and use of a virtual learning environment in China. Computers & Education, 50(3), 838-852.
76. Vaz, M. S. M. G., de Bittencourt, D. F., Vaz, M. C. S., & Júnior, A. S. (2015). Information technology and communication and educational practices. Iberoamerican Journal of Applied Computing, 5(1), 7-15.
77. Vuopala, E., Hyvönen, P., Järvelä, S. (2016). Interaction forms in successful collaborative learning in virtual learning environments. Active Learning in Higher Education, 17(1), 25–38.
78. Wang, Y. S. (2003). Assessment of learner satisfaction with asynchronous electronic learning systems. Information & Management, 41, 75–86.
79. Yilmaz, R. M., Karaman, A., Karakuş, T., & Goktas, Y. (2014). İlköğretim öğrencilerinin 3 boyutlu sanal öğrenme ortamlarına yönelik tutumları: Second life örneği. Ege Eğitim Dergisi, 15(2), 538-555.
80. Yukselturk, E., & Bulut, S. (2009). Gender differences in self-regulated online learning environment. Journal of Educational Technology & Society, 12(3), 12-22.
81. Zafra, A., Gibaja, E., Luque, M., Ventura, S. (2011). An Evaluation of the Effectiveness of e-Learning System as Support for Traditional Classes. 7th International Conference on Next Generation Web Services Practices. October 19-21, 2011. Salamanca, Spain.
82. Zerihun, Z., Beishuizen., J., & Os, W. (2012) Student learning experience as indicator of teaching quality. Educational Assessment, Evaluation and Accountability, 24(2), 99-111.
83. Zhang, D., Zhao, J. L., Zhou, L., & Nunamaker Jr, J. F. (2004). Can e-learning replace classroom learning?. Communications of the ACM, 47(5), 75-79.
84. Zhu, C. (2012). Student satisfaction, performance and knowledge construction in online collaborative learning. Journal of Educational Technology and Society, 15(1), 127-136.