Development of an online evaluation awareness scale

Abstract

The COVID-19 epidemic has precipitated a rapid and widespread adoption of online education, leading to its normalization in contemporary society. Online education is evident across several educational levels. However, assessing the efficacy and effectiveness of these training programs can only be achieved by implementing a suitable evaluation methodology. One of the primary challenges associated with online education is the difficulty in assessing its quality and effectiveness. One of the contributing factors to this issue is the instructor's lack of technological skills and knowledge relevant to online teaching. This research aims to develop a scale (Online Evaluation Awareness-OnEvA) to determine the instructors’ awareness of online evaluation. 165 participants’ data from 63 universities was used for exploratory factor analysis. The items of the scale are designed to measure awareness in both pedagogical and technological dimensions. The items in the pedagogical dimension include the essential competencies that instructors should possess, and alternative evaluation methods offered online. In the technological dimension, the items are created to determine instructors’ awareness of the use of technological platforms and tools. For the next step, a confirmatory factor analysis was performed to establish the scale’s construct validity with 161 instructors. Additionally, the scale’s internal consistency was determined to be high, with a Cronbach's alpha of .964. It is affirmed that this awareness scale developed within the scope of this research will contribute to the development of future models or frameworks related to the dimensions of evaluation knowledge in online learning.

Keywords

• online learning
• awareness
• online evaluation
• scale development

References

1. Ali, K., Barhom, N., & Duggal, M. S. (2022). Online assessment platforms: What is on offer? European Journal of Dental Education. doi:10.1111/eje.12807
2. Al Roomy, M. A. (2022). Exploring Teacher’s Perspectives about Online Assessment during the COVID-19 Pandemic in a Saudi Context. Arab World English Journal, 8, 3–16. https://doi.org/10.24093/awej/call8.1
3. Appelbaum, S. A. (1973) Psychological-Mindedness: Word, Concept and Essence. International Journal of Psychoanalysis, 54, 35-46.
4. Arifin, Z., & Setiawan, B. (2022). Utilising gamification for online evaluation through Quizizz: Teachers’ perspectives and experiences. World Journal on Educational Technology: Current Issues, 14(3), 781–796. https://doi.org/10.18844/wjet.v14i3.7278
5. Arnold, I. J. M. (2016). Cheating at online formative tests: Does it pay off? The Internet and Higher Education, 29, 98–106. doi:10.1016/j.iheduc.2016.02.001
6. Attali, Y., & Arieli-Attali, M. (2015). Gamification in assessment: Do points affect test performance? Computers and Education, 83, 57–63. doi:10.1016/j.compedu.2014.12.012
7. Balçın, M., & Ergün, A. (2016). Technological pedagogical content knowledge (TPCK) self-efficacy scale for pre-service science teachers on material development: development, reliability and validity study . Turkish Journal of Education, 5(3), 130-143.
8. Bozkurt, A., & Sharma, R. (2020). Emergency remote teaching in a time of global crisis due to CoronaVirus pandemic. Asian Journal of Distance Education, 15(1), 1–6.

9. Brown, R. F., John, A. S., Hu, Y., & Sandhu, G. (2024). Differential Electronic Survey Response: Does Survey Fatigue Affect Everyone Equally?. Journal of Surgical Research, 294, 191-197.
10. Burnham, K. D., Major, C. A., & Borman, W. H. (2023). First-person video experiences as a vicarious, virtual alternative to in-person basic science labs. Journal of Chiropractic Education, 37(1), 7-12.
11. Büyüköztürk, Ş., Akgün, Ö. E., Özkahveci, Ö. ve Demirel, F. (2004). Güdülenme ve öğrenme stratejileri ölçeğinin türkçe formunun geçerlik ve güvenirlik çalışması [The Validity and Reliability Study of the Turkish Version of the Motivated Strategies for Learning Questionnaire] Kuram ve Uygulamada Eğitim Bilimleri, 4(2), 207-239.
12. Büyüköztürk, Ş. (2014). Sosyal bilimler için veri analizi el kitabı [Manual of data analysis for social sciences]. Ankara: Pegem Akademi.
13. Byrne, V. L., Hogan, E., Dhingra, N., Anthony, M., & Gannon, C. (2021). An exploratory study of how novice instructors pivot to online assessments strategies. Distance Education, 42(2), 184–199. https://doi.org/10.1080/01587919.2021.1911624
14. Cakir, H. (2013). Use of blogs in pre-service teacher education to improve student engagement. Computers and Education, 68, 244–252. doi:10.1016/j.compedu.2013.05.013
15. Canbazoğlu-Bilici, S., Yamak, H., Kavak, N., & Guzey, S. S. (2013). Technological Pedagogical Content Knowledge Self-Efficacy Scale (TPACK-SeS) for pre-service science teachers: Construction, validation, and reliability. Eurasian Journal of Educational Research, 52, 37-60.
16. Chai, C. S., Tan, L., Deng, F., & Koh, J. H. L. (2017). Examining pre-service teachers’ design capacities for web-based 21st century new culture of learning. Australasian Journal of Educational Technology, 33(2), 129-142.
17. Chang, C.-C., Tseng, K.-H., & Liang, C. (2014). Is Reflection Performance Correlated to the Learning Effect in a Web-Based Portfolio Assessment Environment for Middle School Students? Asia-Pacific Education Researcher, 23(1), 73–82. doi:10.1007/s40299-013-0087-8
18. Cheek, D. (2022). Digital transformation and online learning within the context of the future of higher education. Journal of Educational Technology and Online Learning, 5(4), 781–794. https://doi.org/10.31681/jetol.1147713
19. Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk Ş. (2012). Sosyal Bilimler İçin Çok Değişkenli İstatistik SPSS ve LISREL Uygulamaları [Multivariate Statistics SPSS and LISREL Applications for Social Sciences] Ankara: Pegem.
20. Fornell, C. and Larcker, D.F. (1981) Evaluating Structural Equation Models with Unobservable Variables and Measurement Error. Journal of Marketing Research, 18, 39-50. https://doi.org/10.2307/3151312.
21. Francom, G. M., Lee, S. J., & Pinkney, H. (2021). Technologies, challenges and needs of K-12 teachers in the transition to distance learning during the COVID-19 pandemic. TechTrends, 65(4), 589-601.
22. Ferretti, F., Santi, G. R. P., Del Zozzo, A., Garzetti, M., & Bolondi, G. (2021). Assessment Practices and Beliefs: Teachers’ Perspectives on Assessment during Long Distance Learning. Education and Sciences, 11(6). https://doi.org/10.3390/educsci11060264
23. García-Peñalvo, F. J., Corell, A., Abella-García, V., & Grande-de-Prado, M. (2021). Recommendations for Mandatory Online Assessment in Higher Education During the COVID-19 Pandemic. In D. Burgos, A. Tlili ve A. Tabacco (Eds.), Radical Solutions for Education in a Crisis Context: COVID-19 as an Opportunity for Global Learning (pp. 85–98). Singapore: Springer.
24. Gaytan, J., & McEwen, B. C. (2007). Effective online instructional and assessment strategies. International Journal of Phytoremediation, 21(1), 117–132. doi:10.1080/08923640701341653
25. Gikandi, J. W., Morrow, D., & Davis, N. E. (2011). Online formative assessment in higher education: A review of the literature. Computers and Education, 57(4), 2333–2351. doi:10.1016/j.compedu.2011.06.004
26. Ferretti, F., Santi, G. R. P., Del Zozzo, A., Garzetti, M., & Bolondi, G. (2021). Assessment Practices and Beliefs: Teachers’ Perspectives on Assessment during Long Distance Learning. Education and Sciences, 11(6). https://doi.org/10.3390/educsci11060264
27. Hacıömeroğlu, G., Yaşar, C., & Ökmen, A. K. (2018). Teknoloji Destekli Sınıf Ortamlarında Öğrencilerin Ders Sorumlusunun Bilgisine Yönelik Algıları Ölçeği geçerlik ve güvenirlik çalışması [A Validity And Reliability Study of Students’ Perceptions of Faculty Knowledge (SPFK) in Technology Supported Environments]. Ege Eğitim Dergisi, 19(1), 338-352.
28. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. doi:10.3102/003465430298487
29. Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1), 115-135. doi: 10.1007/s11747-014-0403-8
30. Hiçyılmaz, Y. (2018). Görsel sanatlar öğretmen adaylarının teknolojik pedagojik içerik bilgisi öz yeterlikleri [Self-efficacy of technical pedagogical content knowledge of visual arts teacher candidates]. Doctoral dissertation, Ondokuz Mayıs University Institute of Educational Sciences, Samsun.
31. Hsu, C. Y., Tsai, M. J., Chang, Y. H., & Liang, J. C. (2017). Surveying in-service teachers’ beliefs about game-based learning and perceptions of technological pedagogical and content knowledge of games. Journal of Educational Technology & Society, 20(1), 134-143.
32. İzmirli, S., & Kırmacı, Ö. (2017). Developing Online Teaching Competencies of Educators in Turkey. Mediterranean Journal of Educational Research, 22, 38–52.
33. Johnson, M. (2016). Feedback effectiveness in professional learning contexts. Review of Education, 4(2), 195–229. https://doi.org/10.1002/rev3.3061
34. Junus, K., Santoso, H. B., Putra, P. O. H., Gandhi, A., & Siswantining, T. (2021). Lecturer readiness for online classes during the pandemic: A survey research. Education sciences, 11(3), 139.
35. Kalaycı, Ş. (2010). SPSS Uygulamalı Çok Değişkenli İstatistik Teknikleri [SPSS Applied Multivariate Statistical Techniques]. Ankara: Asil.
36. Karaman, H., Burcu, A. T. A. R., & Aktan, D. Ç. (2017). Açımlayıcı faktör analizinde kullanılan faktör çıkartma yöntemlerinin karşılaştırılması. Gazi Üniversitesi Gazi Eğitim Fakültesi Dergisi, 37(3), 1173-1193.
37. Kartal, T., Kartal, B., & Uluay, G. (2016). Öğretmen adayları için teknolojik pedagojik alan bilgisi öz-değerlendirme ölçeği: geliştirilmesi, geçerlik ve güvenirlik çalışması [Technological pedagogical content knowledge self-assessment scale for pre-service teachers: development, validity and reliability study]. Uluslararası Avrasya Sosyal Bilimler Dergisi, 7(23), 1-36.
38. Kaya, S. , & Dağ, F. (2013). Sınıf Öğretmenlerine Yönelik Teknolojik Pedagojik İçerik Bilgisi Ölçeği’nin Türkçe’ye Uyarlanması [Turkish Adaptation of Technological Pedagogical Content Knowledge Survey for Elementary Teachers] Kuram ve Uygulamada Eğitim Bilimleri, 13(1), 291-306.
39. Kaya, Z., Kaya, O. N., & Emre, İ. (2013). Teknolojik Pedagojik Alan Bilgisi (TPAB) Ölçeği’nin Türkçe’ye uyarlanması [Adaptation of Technological Pedagogical Content Knowledge Scale to Turkish]. Kuram ve Uygulamada Eğitim Bilimleri, 13(4), 2355-2377.
40. Keldgord, F., & Ching, Y. H. (2022). Teachers’ Experiences with and Perceptions of Virtual Manipulatives Following the COVID-19 Pandemic. TechTrends, 66(6), 957-967.
41. King, C., & Guyette, G. (2009). Online Exams and Cheating: An Empirical Analysis of Business Students’ Views. Journal of Educators Online, 6(1), 1-11.
42. Kirkan, B., & Kalelioğlu, F. (2017). Türkiye ’ de Uzaktan Eğitim Merkezlerinin Durumu: Betimsel Bir Çalışma [The Situation of Distance Education Centers in Turkey: A Descriptive Study] Journal of Instructional Technologies & Teacher Education, 6(3), 88–98.
43. Kırmacı, Ö. (2022). Uzaktan eğitimde oyunlaştırılmış öğrenme ortamının tasarlanması ve geliştirilmesi [Design and development of gamified online learning environment in distance education]. Doctoral Dissertation, Gazi University Institute of Educational Sciences, Ankara.
44. Kurnaz, F. B. (2021). Online peer assessment in teacher education. Journal of Educational Technology and Online Learning, 4(4), 835–853. https://doi.org/10.31681/jetol.987902
45. Lawshe, C. H. (1975). A Quantitative Approach To Content Validity. Personnel Psychology, 28, 563-575.
46. Layco, E. P., Parico, A. D., Magno, J. C., Honorio, D., & State, V. (2022). Probing Teachers` Competencies In Using Online Assessment Tools : Basis For A Capacity Building Program. International Journal of Early Childhood Special Education, 14(3), 5143–5157. doi:10.9756/INT-JECSE/V14I3.676
47. Lee, S. J., & Kwon, K. (2021). Peer Assessment as a Facilitating and Assessment Strategy in Online and Face-to-Face Classes. International Journal oOf Online Pedagogy And Course Design, 11(3), 36–48. doi:10.4018/IJOPCD.2021070103
48. Li, H., Xiong, Y., Hunter, C. V., Guo, X., & Tywoniw, R. (2020). Does peer assessment promote student learning? A meta-analysis. Assessment and Evaluation in Higher Education, 45(2), 193–211. doi:10.1080/02602938.2019.1620679
49. Loureiro, P., & Gomes, M. J. (2023). Online Peer Assessment for Learning: Findings from Higher Education Students. Education Sciences, 13(3). https://doi.org/10.3390/educsci13030253
50. Meccawy, Z., Meccawy, M., & Alsobhi, A. (2021). Assessment in ‘survival mode’: student and faculty perceptions of online assessment practices in HE during Covid-19 pandemic. International Journal for Educational Integrity, 17(1), 1–24. doi:10.1007/s40979-021-00083-9
51. Meek, S. E. M. M., Blakemore, L., & Marks, L. (2017). Is peer review an appropriate form of assessment in a MOOC? Student participation and performance in formative peer review. Assessment and Evaluation in Higher Education, 42(6), 1000–1013. doi:10.1080/02602938.2016.1221052
52. Mirza, H. S. (2021). University Teachers ’ Perceptions of Online Assessment during the Covid-19 Pandemic in Lebanon. American Academic & Scholarly Research Journal, 13(1), 11–24.
53. Moccozet, L., Tardy, C., Opprecht, W., & Leonard, M. (2013, September). Gamification-based assessment of group work. Paper presented at 2013 International Conference on Interactive Collaborative Learning, 171–179. doi:10.1109/ICL.2013.6644565
54. Ng, W. S., Xie, H., & Wang, F. L. (2018). Enhancing Teacher Assessment Literacy Using a Blended Deep Learning Approach. In S. K. S. Cheung, L. Kwok, K. Kubota, L. K. Lee, J. Tokito (Eds.) Blended learning: Enhancing learning success (pp. 203–214). doi:10.1007/978-3-319-94505-7_16
55. Okada, A., Whitelock, D., Holmes, W., & Edwards, C. (2019). e-Authentication for online assessment: A mixed-method study. British Journal of Educational Technology, 50(2), 861–875. doi:10.1111/bjet.12608
56. Okur, M. R., Kılınç, H., & Usta, İ. (2021). The Opinions of Field Experts on Online Test Applications and Test Security During the COVID-19 Pandemic. International Journal of Assessment Tools in Education, 8(4), 975–990. doi:10.21449/ijate.875293
57. Önal, N. (2016). Development, Validity and Reliability of TPACK Scale with Pre-Service Mathematics Teachers. International Online Journal of Educational Sciences, 8(2), 93-107.
58. Öztürk, E., & Horzum, M. B. (2011). Teknolojik Pedagojik İçerik Bilgisi Ölçeği’nin Türkçeye uyarlaması [Adaptation of Technological Pedagogical Content Knowledge Scale to Turkish]. Ahi Evran Üniversitesi Eğitim Fakültesi Dergisi, 12(3), 255-278.
59. Öztürk, M. (2021). Asynchronous online learning experiences of students in pandemic process: Facilities, challenges, suggestions. Turkish Online Journal of Qualitative Inquiry, 12(2), 173-200.
60. Pulham, E., & Graham, C. R. (2018). Comparing K-12 online and blended teaching competencies: A literature review. Distance Education, 39(3), 411-432.
61. Şahin, N. H., & Yeniçeri, Z. (2015). “Farkındalık” Üzerine Üç Araç: Psikolojik Farkındalık, Bütünleyici Kendilik Farkındalık ve Toronto Bilgece Farkındalık Ölçekleri. [Three Assessment Scales on Awareness: Psychological Mindedness, Integrative Self Knowledge and Toronto Trait Mindfulness Scales].Türk Psikoloji Dergisi, 30(76), 48–67.
62. Seçer, İ. (2013). SPSS ve LISREL ile Pratik Veri Analizi. [Practical Data Analysis with SPSS and LISREL.] Ankara: Anı.
63. Seifert, T., & Feliks, O. (2019). Online self-assessment and peer-assessment as a tool to enhance student-teachers’ assessment skills. Assessment & Evaluation in Higher Education, 44(2), 169–185. doi:10.1080/02602938.2018.1487023
64. Şenel, S., Pekdağ, B. & Sarıtaş, M. T. (2018). Turkish Adaptation of Instrument for Assessing Undergraduate Students’ Perceptions of Faculty Knowledge in Technology-Supported Class Environments . Bartın University Journal of Faculty of Education, 7(1), 67-93. doi: 10.14686/buefad.363938
65. Shi J, Mo X, & Sun Z. (2012) Content validity index in scale development. doi: 10.3969/j.issn 1672-7347 .2012.02.007.
66. Shin, M., & Hickey, K. (2021). Needs a little TLC: Examining college students’ emergency remote teaching and learning experiences during COVID-19. Journal of Further and Higher Education, 45(7), 973-986.
67. Stein, D. S. ve Wanstreet, C. E. (2015). Discussions in online and blended learning: A Tool for peer assessment. In S. Koç, X. Liu, & P. Wachira (Eds.), Assessment in online and blended learning environments (pp. 253–267). Charlotte: Information Age.
68. Sun-Lin, H.-Z., & Chiou, G.-F. (2019). Effects of Gamified Comparison on Sixth Graders’ Algebra Word Problem Solving and Learning Attitude. Educational Technology and Society, 22(1), 120–130.
69. Tabachnick, B. G., & Fidell, L. S. (2007). Using Multivariate Statistics. London: Pearson.
70. Taşkın, N., & Kılıç Çakmak, E. (2017). The Use of Gamification in Student Centered Learning Environments as Alternative Assessment. Bartın University Journal of Faculty of Education, 6 (3) , 1227-1248. DOI: 10.14686/buefad.333286
71. Timur, B., & Taşar, M. F. (2011). The Adaptation of the Technological Pedagogical Content Knowledge Confidence Survey into Turkish. Gaziantep University Journal of Social Sciences, 10(2) , 839-856.
72. Topuz, A. C., Saka, E., Fatsa, Ö. F., & Kurşun, E. (2022). Emerging trends of online assessment systems in the emergency remote teaching period. Smart Learning Environments, 9(1), 1-21. doi:10.1186/s40561-022-00199-6
73. Trullen, J., & Rodríguez, S. (2013). Faculty perceptions of instrumental and improvement reasons behind quality assessments in higher education: the roles of participation and identification. Studies in Higher Education, 38(5), 678–692. doi:10.1080/03075079.2011.590587
74. Valtonen, T., Sointu, E., Kukkonen, J., Kontkanen, S., Lambert, M. C., & Mäkitalo-Siegl, K. (2017). TPACK updated to measure pre-service teachers’ twenty-first century skills. Australasian Journal of Educational Technology, 33(3), 15-31.
75. Van Schoors, R., Elen, J., Raes, A., & Depaepe, F. (2021). An overview of 25 years of research on digital personalized learning in primary and secondary education: A systematic review of conceptual and methodological trends. British Journal of Educational Technology, 52(5), 1798–1822. https://doi.org/10.1111/bjet.13148
76. Van Maele, J., Baten, L., Beaven, A., & Rajagopal, K. (2013). e-Assessment for Learning: Gaining Insight in Language Learning with Online Assessment Environments. In Computer-assisted foreign language teaching and learning: technological advances (pp. 245–261). Hershey: IGI Global.
77. Villiers, R. de, Scott-Kennel, J., & Larke, R. (2016). Principles of Effective E-Assessment: A Proposed Framework. Journal of International Business Education, 11, 65–92.
78. Vonderwell, S. K., & Boboc, M. (2013). Promoting Formative Assessment in Online Teaching and Learning. TechTrends, 57(4), 22–27. doi:10.1007/s11528-013-0673-x
79. Watson, S. L., & Reigeluth, C. M. (2008). The learner-centered paradigm of education. Educational Technology, 4, 1–32. doi:10.4324/9781315795478
80. Xiong, Y., & Suen, H. K. (2018). Assessment approaches in massive open online courses: Possibilities, challenges and future directions. International Review of Education, 64(2, SI), 241–263. doi:10.1007/s11159-018-9710-5
81. Yurdakul, I. K., Odabasi, H. F., Kilicer, K., Coklar, A. N., Birinci, G., & Kurt, A. A. (2012). The development, validity and reliability of TPACK-deep: A technological pedagogical content knowledge scale. Computers & Education, 58(3), 964-977.
82. Yüksek Öğretim Kurumu (2022). Yükseköğretim Kurumlarında Uzaktan Öğreti̇me İli̇şki̇n Usul ve Esaslar. [Procedures and Principles Regarding Distance Education in Higher Education Institutions.] Retrieved from https://www.yok.gov.tr/Documents/Kurumsal/egitim_ogretim_dairesi/Uzaktan_ogretim/yuksekogretim-kurumlarinda-uzaktan-ogretime-iliskin-usul-ve-esaslar.pdf
83. Yüksek Öğretim Kurumu (2020). Küresel Salgında Yeni Normalleşme Süreci. [New Normalization Process in the Global Pandemic.] Retrieved from https://www.yok.gov.tr/Documents/Yayinlar/Yayinlarimiz/2020/kuresel-salginda-yeni-normallesme-sureci-2020.pdf
84. Zamanzadeh, V., Ghahramanian, A., Rassouli, M., Abbaszadeh, A., Alavi-Majd, H., & Nikanfar, A. R. (2015). Design and implementation content validity study: Development of an instrument for measuring patient-centered communication. Journal of caring sciences, 4(2), 165–178. https://doi.org/10.15171/jcs.2015.017
85. Zenouzagh, Z. M. (2019). The effect of online summative and formative teacher assessment on teacher competences. Asia Pacific Education Review, 20(3), 343–359. doi:10.1007/s12564-018-9566-1
86. Zheng, L., Chen, N.-S., Cui, P., & Zhang, X. (2019). A Systematic Review of Technology-Supported Peer Assessment Research: An Activity Theory Approach. International Review of Research in Open and Distributed Learning, 20(5), 168–191.