Development of an online evaluation awareness scale

Year 2024, Volume: 7 Issue: 1, 34 - 51, 31.01.2024

Pınar Mıhcı Türker , Ömer Kırmacı , Emrah Kayabaşı , Erinç Karataş , Ebru Kılıç Çakmak , Serçin Karataş

https://doi.org/10.31681/jetol.1265987

Abstract

The COVID-19 epidemic has precipitated a rapid and widespread adoption of online education, leading to its normalization in contemporary society. Online education is evident across several educational levels. However, assessing the efficacy and effectiveness of these training programs can only be achieved by implementing a suitable evaluation methodology. One of the primary challenges associated with online education is the difficulty in assessing its quality and effectiveness. One of the contributing factors to this issue is the instructor's lack of technological skills and knowledge relevant to online teaching. This research aims to develop a scale (Online Evaluation Awareness-OnEvA) to determine the instructors’ awareness of online evaluation. 165 participants’ data from 63 universities was used for exploratory factor analysis. The items of the scale are designed to measure awareness in both pedagogical and technological dimensions. The items in the pedagogical dimension include the essential competencies that instructors should possess, and alternative evaluation methods offered online. In the technological dimension, the items are created to determine instructors’ awareness of the use of technological platforms and tools. For the next step, a confirmatory factor analysis was performed to establish the scale’s construct validity with 161 instructors. Additionally, the scale’s internal consistency was determined to be high, with a Cronbach's alpha of .964. It is affirmed that this awareness scale developed within the scope of this research will contribute to the development of future models or frameworks related to the dimensions of evaluation knowledge in online learning.

Keywords

online learning, awareness, online evaluation, scale development

References

• Ali, K., Barhom, N., & Duggal, M. S. (2022). Online assessment platforms: What is on offer? European Journal of Dental Education. doi:10.1111/eje.12807
• Al Roomy, M. A. (2022). Exploring Teacher’s Perspectives about Online Assessment during the COVID-19 Pandemic in a Saudi Context. Arab World English Journal, 8, 3–16. https://doi.org/10.24093/awej/call8.1
• Appelbaum, S. A. (1973) Psychological-Mindedness: Word, Concept and Essence. International Journal of Psychoanalysis, 54, 35-46.
• Arifin, Z., & Setiawan, B. (2022). Utilising gamification for online evaluation through Quizizz: Teachers’ perspectives and experiences. World Journal on Educational Technology: Current Issues, 14(3), 781–796. https://doi.org/10.18844/wjet.v14i3.7278
• Arnold, I. J. M. (2016). Cheating at online formative tests: Does it pay off? The Internet and Higher Education, 29, 98–106. doi:10.1016/j.iheduc.2016.02.001
• Attali, Y., & Arieli-Attali, M. (2015). Gamification in assessment: Do points affect test performance? Computers and Education, 83, 57–63. doi:10.1016/j.compedu.2014.12.012
• Balçın, M., & Ergün, A. (2016). Technological pedagogical content knowledge (TPCK) self-efficacy scale for pre-service science teachers on material development: development, reliability and validity study . Turkish Journal of Education, 5(3), 130-143.
• Bozkurt, A., & Sharma, R. (2020). Emergency remote teaching in a time of global crisis due to CoronaVirus pandemic. Asian Journal of Distance Education, 15(1), 1–6.
• Brown, R. F., John, A. S., Hu, Y., & Sandhu, G. (2024). Differential Electronic Survey Response: Does Survey Fatigue Affect Everyone Equally?. Journal of Surgical Research, 294, 191-197.
• Burnham, K. D., Major, C. A., & Borman, W. H. (2023). First-person video experiences as a vicarious, virtual alternative to in-person basic science labs. Journal of Chiropractic Education, 37(1), 7-12.
• Büyüköztürk, Ş., Akgün, Ö. E., Özkahveci, Ö. ve Demirel, F. (2004). Güdülenme ve öğrenme stratejileri ölçeğinin türkçe formunun geçerlik ve güvenirlik çalışması [The Validity and Reliability Study of the Turkish Version of the Motivated Strategies for Learning Questionnaire] Kuram ve Uygulamada Eğitim Bilimleri, 4(2), 207-239.
• Büyüköztürk, Ş. (2014). Sosyal bilimler için veri analizi el kitabı [Manual of data analysis for social sciences]. Ankara: Pegem Akademi.
• Byrne, V. L., Hogan, E., Dhingra, N., Anthony, M., & Gannon, C. (2021). An exploratory study of how novice instructors pivot to online assessments strategies. Distance Education, 42(2), 184–199. https://doi.org/10.1080/01587919.2021.1911624
• Cakir, H. (2013). Use of blogs in pre-service teacher education to improve student engagement. Computers and Education, 68, 244–252. doi:10.1016/j.compedu.2013.05.013
• Canbazoğlu-Bilici, S., Yamak, H., Kavak, N., & Guzey, S. S. (2013). Technological Pedagogical Content Knowledge Self-Efficacy Scale (TPACK-SeS) for pre-service science teachers: Construction, validation, and reliability. Eurasian Journal of Educational Research, 52, 37-60.
• Chai, C. S., Tan, L., Deng, F., & Koh, J. H. L. (2017). Examining pre-service teachers’ design capacities for web-based 21st century new culture of learning. Australasian Journal of Educational Technology, 33(2), 129-142.
• Chang, C.-C., Tseng, K.-H., & Liang, C. (2014). Is Reflection Performance Correlated to the Learning Effect in a Web-Based Portfolio Assessment Environment for Middle School Students? Asia-Pacific Education Researcher, 23(1), 73–82. doi:10.1007/s40299-013-0087-8
• Cheek, D. (2022). Digital transformation and online learning within the context of the future of higher education. Journal of Educational Technology and Online Learning, 5(4), 781–794. https://doi.org/10.31681/jetol.1147713
• Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk Ş. (2012). Sosyal Bilimler İçin Çok Değişkenli İstatistik SPSS ve LISREL Uygulamaları [Multivariate Statistics SPSS and LISREL Applications for Social Sciences] Ankara: Pegem.
• Fornell, C. and Larcker, D.F. (1981) Evaluating Structural Equation Models with Unobservable Variables and Measurement Error. Journal of Marketing Research, 18, 39-50. https://doi.org/10.2307/3151312.
• Francom, G. M., Lee, S. J., & Pinkney, H. (2021). Technologies, challenges and needs of K-12 teachers in the transition to distance learning during the COVID-19 pandemic. TechTrends, 65(4), 589-601.
• Ferretti, F., Santi, G. R. P., Del Zozzo, A., Garzetti, M., & Bolondi, G. (2021). Assessment Practices and Beliefs: Teachers’ Perspectives on Assessment during Long Distance Learning. Education and Sciences, 11(6). https://doi.org/10.3390/educsci11060264
• García-Peñalvo, F. J., Corell, A., Abella-García, V., & Grande-de-Prado, M. (2021). Recommendations for Mandatory Online Assessment in Higher Education During the COVID-19 Pandemic. In D. Burgos, A. Tlili ve A. Tabacco (Eds.), Radical Solutions for Education in a Crisis Context: COVID-19 as an Opportunity for Global Learning (pp. 85–98). Singapore: Springer.
• Gaytan, J., & McEwen, B. C. (2007). Effective online instructional and assessment strategies. International Journal of Phytoremediation, 21(1), 117–132. doi:10.1080/08923640701341653
• Gikandi, J. W., Morrow, D., & Davis, N. E. (2011). Online formative assessment in higher education: A review of the literature. Computers and Education, 57(4), 2333–2351. doi:10.1016/j.compedu.2011.06.004
• Ferretti, F., Santi, G. R. P., Del Zozzo, A., Garzetti, M., & Bolondi, G. (2021). Assessment Practices and Beliefs: Teachers’ Perspectives on Assessment during Long Distance Learning. Education and Sciences, 11(6). https://doi.org/10.3390/educsci11060264
• Hacıömeroğlu, G., Yaşar, C., & Ökmen, A. K. (2018). Teknoloji Destekli Sınıf Ortamlarında Öğrencilerin Ders Sorumlusunun Bilgisine Yönelik Algıları Ölçeği geçerlik ve güvenirlik çalışması [A Validity And Reliability Study of Students’ Perceptions of Faculty Knowledge (SPFK) in Technology Supported Environments]. Ege Eğitim Dergisi, 19(1), 338-352.
• Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. doi:10.3102/003465430298487
• Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1), 115-135. doi: 10.1007/s11747-014-0403-8
• Hiçyılmaz, Y. (2018). Görsel sanatlar öğretmen adaylarının teknolojik pedagojik içerik bilgisi öz yeterlikleri [Self-efficacy of technical pedagogical content knowledge of visual arts teacher candidates]. Doctoral dissertation, Ondokuz Mayıs University Institute of Educational Sciences, Samsun.
• Hsu, C. Y., Tsai, M. J., Chang, Y. H., & Liang, J. C. (2017). Surveying in-service teachers’ beliefs about game-based learning and perceptions of technological pedagogical and content knowledge of games. Journal of Educational Technology & Society, 20(1), 134-143.
• İzmirli, S., & Kırmacı, Ö. (2017). Developing Online Teaching Competencies of Educators in Turkey. Mediterranean Journal of Educational Research, 22, 38–52.
• Johnson, M. (2016). Feedback effectiveness in professional learning contexts. Review of Education, 4(2), 195–229. https://doi.org/10.1002/rev3.3061
• Junus, K., Santoso, H. B., Putra, P. O. H., Gandhi, A., & Siswantining, T. (2021). Lecturer readiness for online classes during the pandemic: A survey research. Education sciences, 11(3), 139.
• Kalaycı, Ş. (2010). SPSS Uygulamalı Çok Değişkenli İstatistik Teknikleri [SPSS Applied Multivariate Statistical Techniques]. Ankara: Asil.
• Karaman, H., Burcu, A. T. A. R., & Aktan, D. Ç. (2017). Açımlayıcı faktör analizinde kullanılan faktör çıkartma yöntemlerinin karşılaştırılması. Gazi Üniversitesi Gazi Eğitim Fakültesi Dergisi, 37(3), 1173-1193.
• Kartal, T., Kartal, B., & Uluay, G. (2016). Öğretmen adayları için teknolojik pedagojik alan bilgisi öz-değerlendirme ölçeği: geliştirilmesi, geçerlik ve güvenirlik çalışması [Technological pedagogical content knowledge self-assessment scale for pre-service teachers: development, validity and reliability study]. Uluslararası Avrasya Sosyal Bilimler Dergisi, 7(23), 1-36.
• Kaya, S. , & Dağ, F. (2013). Sınıf Öğretmenlerine Yönelik Teknolojik Pedagojik İçerik Bilgisi Ölçeği’nin Türkçe’ye Uyarlanması [Turkish Adaptation of Technological Pedagogical Content Knowledge Survey for Elementary Teachers] Kuram ve Uygulamada Eğitim Bilimleri, 13(1), 291-306.
• Kaya, Z., Kaya, O. N., & Emre, İ. (2013). Teknolojik Pedagojik Alan Bilgisi (TPAB) Ölçeği’nin Türkçe’ye uyarlanması [Adaptation of Technological Pedagogical Content Knowledge Scale to Turkish]. Kuram ve Uygulamada Eğitim Bilimleri, 13(4), 2355-2377.
• Keldgord, F., & Ching, Y. H. (2022). Teachers’ Experiences with and Perceptions of Virtual Manipulatives Following the COVID-19 Pandemic. TechTrends, 66(6), 957-967.
• King, C., & Guyette, G. (2009). Online Exams and Cheating: An Empirical Analysis of Business Students’ Views. Journal of Educators Online, 6(1), 1-11.
• Kirkan, B., & Kalelioğlu, F. (2017). Türkiye ’ de Uzaktan Eğitim Merkezlerinin Durumu: Betimsel Bir Çalışma [The Situation of Distance Education Centers in Turkey: A Descriptive Study] Journal of Instructional Technologies & Teacher Education, 6(3), 88–98.
• Kırmacı, Ö. (2022). Uzaktan eğitimde oyunlaştırılmış öğrenme ortamının tasarlanması ve geliştirilmesi [Design and development of gamified online learning environment in distance education]. Doctoral Dissertation, Gazi University Institute of Educational Sciences, Ankara.
• Kurnaz, F. B. (2021). Online peer assessment in teacher education. Journal of Educational Technology and Online Learning, 4(4), 835–853. https://doi.org/10.31681/jetol.987902
• Lawshe, C. H. (1975). A Quantitative Approach To Content Validity. Personnel Psychology, 28, 563-575.
• Layco, E. P., Parico, A. D., Magno, J. C., Honorio, D., & State, V. (2022). Probing Teachers` Competencies In Using Online Assessment Tools : Basis For A Capacity Building Program. International Journal of Early Childhood Special Education, 14(3), 5143–5157. doi:10.9756/INT-JECSE/V14I3.676
• Lee, S. J., & Kwon, K. (2021). Peer Assessment as a Facilitating and Assessment Strategy in Online and Face-to-Face Classes. International Journal oOf Online Pedagogy And Course Design, 11(3), 36–48. doi:10.4018/IJOPCD.2021070103
• Li, H., Xiong, Y., Hunter, C. V., Guo, X., & Tywoniw, R. (2020). Does peer assessment promote student learning? A meta-analysis. Assessment and Evaluation in Higher Education, 45(2), 193–211. doi:10.1080/02602938.2019.1620679
• Loureiro, P., & Gomes, M. J. (2023). Online Peer Assessment for Learning: Findings from Higher Education Students. Education Sciences, 13(3). https://doi.org/10.3390/educsci13030253
• Meccawy, Z., Meccawy, M., & Alsobhi, A. (2021). Assessment in ‘survival mode’: student and faculty perceptions of online assessment practices in HE during Covid-19 pandemic. International Journal for Educational Integrity, 17(1), 1–24. doi:10.1007/s40979-021-00083-9
• Meek, S. E. M. M., Blakemore, L., & Marks, L. (2017). Is peer review an appropriate form of assessment in a MOOC? Student participation and performance in formative peer review. Assessment and Evaluation in Higher Education, 42(6), 1000–1013. doi:10.1080/02602938.2016.1221052
• Mirza, H. S. (2021). University Teachers ’ Perceptions of Online Assessment during the Covid-19 Pandemic in Lebanon. American Academic & Scholarly Research Journal, 13(1), 11–24.
• Moccozet, L., Tardy, C., Opprecht, W., & Leonard, M. (2013, September). Gamification-based assessment of group work. Paper presented at 2013 International Conference on Interactive Collaborative Learning, 171–179. doi:10.1109/ICL.2013.6644565
• Ng, W. S., Xie, H., & Wang, F. L. (2018). Enhancing Teacher Assessment Literacy Using a Blended Deep Learning Approach. In S. K. S. Cheung, L. Kwok, K. Kubota, L. K. Lee, J. Tokito (Eds.) Blended learning: Enhancing learning success (pp. 203–214). doi:10.1007/978-3-319-94505-7_16
• Okada, A., Whitelock, D., Holmes, W., & Edwards, C. (2019). e-Authentication for online assessment: A mixed-method study. British Journal of Educational Technology, 50(2), 861–875. doi:10.1111/bjet.12608
• Okur, M. R., Kılınç, H., & Usta, İ. (2021). The Opinions of Field Experts on Online Test Applications and Test Security During the COVID-19 Pandemic. International Journal of Assessment Tools in Education, 8(4), 975–990. doi:10.21449/ijate.875293
• Önal, N. (2016). Development, Validity and Reliability of TPACK Scale with Pre-Service Mathematics Teachers. International Online Journal of Educational Sciences, 8(2), 93-107.
• Öztürk, E., & Horzum, M. B. (2011). Teknolojik Pedagojik İçerik Bilgisi Ölçeği’nin Türkçeye uyarlaması [Adaptation of Technological Pedagogical Content Knowledge Scale to Turkish]. Ahi Evran Üniversitesi Eğitim Fakültesi Dergisi, 12(3), 255-278.
• Öztürk, M. (2021). Asynchronous online learning experiences of students in pandemic process: Facilities, challenges, suggestions. Turkish Online Journal of Qualitative Inquiry, 12(2), 173-200.
• Pulham, E., & Graham, C. R. (2018). Comparing K-12 online and blended teaching competencies: A literature review. Distance Education, 39(3), 411-432.
• Şahin, N. H., & Yeniçeri, Z. (2015). “Farkındalık” Üzerine Üç Araç: Psikolojik Farkındalık, Bütünleyici Kendilik Farkındalık ve Toronto Bilgece Farkındalık Ölçekleri. [Three Assessment Scales on Awareness: Psychological Mindedness, Integrative Self Knowledge and Toronto Trait Mindfulness Scales].Türk Psikoloji Dergisi, 30(76), 48–67.
• Seçer, İ. (2013). SPSS ve LISREL ile Pratik Veri Analizi. [Practical Data Analysis with SPSS and LISREL.] Ankara: Anı.
• Seifert, T., & Feliks, O. (2019). Online self-assessment and peer-assessment as a tool to enhance student-teachers’ assessment skills. Assessment & Evaluation in Higher Education, 44(2), 169–185. doi:10.1080/02602938.2018.1487023
• Şenel, S., Pekdağ, B. & Sarıtaş, M. T. (2018). Turkish Adaptation of Instrument for Assessing Undergraduate Students’ Perceptions of Faculty Knowledge in Technology-Supported Class Environments . Bartın University Journal of Faculty of Education, 7(1), 67-93. doi: 10.14686/buefad.363938
• Shi J, Mo X, & Sun Z. (2012) Content validity index in scale development. doi: 10.3969/j.issn 1672-7347 .2012.02.007.
• Shin, M., & Hickey, K. (2021). Needs a little TLC: Examining college students’ emergency remote teaching and learning experiences during COVID-19. Journal of Further and Higher Education, 45(7), 973-986.
• Stein, D. S. ve Wanstreet, C. E. (2015). Discussions in online and blended learning: A Tool for peer assessment. In S. Koç, X. Liu, & P. Wachira (Eds.), Assessment in online and blended learning environments (pp. 253–267). Charlotte: Information Age.
• Sun-Lin, H.-Z., & Chiou, G.-F. (2019). Effects of Gamified Comparison on Sixth Graders’ Algebra Word Problem Solving and Learning Attitude. Educational Technology and Society, 22(1), 120–130.
• Tabachnick, B. G., & Fidell, L. S. (2007). Using Multivariate Statistics. London: Pearson.
• Taşkın, N., & Kılıç Çakmak, E. (2017). The Use of Gamification in Student Centered Learning Environments as Alternative Assessment. Bartın University Journal of Faculty of Education, 6 (3) , 1227-1248. DOI: 10.14686/buefad.333286
• Timur, B., & Taşar, M. F. (2011). The Adaptation of the Technological Pedagogical Content Knowledge Confidence Survey into Turkish. Gaziantep University Journal of Social Sciences, 10(2) , 839-856.
• Topuz, A. C., Saka, E., Fatsa, Ö. F., & Kurşun, E. (2022). Emerging trends of online assessment systems in the emergency remote teaching period. Smart Learning Environments, 9(1), 1-21. doi:10.1186/s40561-022-00199-6
• Trullen, J., & Rodríguez, S. (2013). Faculty perceptions of instrumental and improvement reasons behind quality assessments in higher education: the roles of participation and identification. Studies in Higher Education, 38(5), 678–692. doi:10.1080/03075079.2011.590587
• Valtonen, T., Sointu, E., Kukkonen, J., Kontkanen, S., Lambert, M. C., & Mäkitalo-Siegl, K. (2017). TPACK updated to measure pre-service teachers’ twenty-first century skills. Australasian Journal of Educational Technology, 33(3), 15-31.
• Van Schoors, R., Elen, J., Raes, A., & Depaepe, F. (2021). An overview of 25 years of research on digital personalized learning in primary and secondary education: A systematic review of conceptual and methodological trends. British Journal of Educational Technology, 52(5), 1798–1822. https://doi.org/10.1111/bjet.13148
• Van Maele, J., Baten, L., Beaven, A., & Rajagopal, K. (2013). e-Assessment for Learning: Gaining Insight in Language Learning with Online Assessment Environments. In Computer-assisted foreign language teaching and learning: technological advances (pp. 245–261). Hershey: IGI Global.
• Villiers, R. de, Scott-Kennel, J., & Larke, R. (2016). Principles of Effective E-Assessment: A Proposed Framework. Journal of International Business Education, 11, 65–92.
• Vonderwell, S. K., & Boboc, M. (2013). Promoting Formative Assessment in Online Teaching and Learning. TechTrends, 57(4), 22–27. doi:10.1007/s11528-013-0673-x
• Watson, S. L., & Reigeluth, C. M. (2008). The learner-centered paradigm of education. Educational Technology, 4, 1–32. doi:10.4324/9781315795478
• Xiong, Y., & Suen, H. K. (2018). Assessment approaches in massive open online courses: Possibilities, challenges and future directions. International Review of Education, 64(2, SI), 241–263. doi:10.1007/s11159-018-9710-5
• Yurdakul, I. K., Odabasi, H. F., Kilicer, K., Coklar, A. N., Birinci, G., & Kurt, A. A. (2012). The development, validity and reliability of TPACK-deep: A technological pedagogical content knowledge scale. Computers & Education, 58(3), 964-977.
• Yüksek Öğretim Kurumu (2022). Yükseköğretim Kurumlarında Uzaktan Öğreti̇me İli̇şki̇n Usul ve Esaslar. [Procedures and Principles Regarding Distance Education in Higher Education Institutions.] Retrieved from https://www.yok.gov.tr/Documents/Kurumsal/egitim_ogretim_dairesi/Uzaktan_ogretim/yuksekogretim-kurumlarinda-uzaktan-ogretime-iliskin-usul-ve-esaslar.pdf
• Yüksek Öğretim Kurumu (2020). Küresel Salgında Yeni Normalleşme Süreci. [New Normalization Process in the Global Pandemic.] Retrieved from https://www.yok.gov.tr/Documents/Yayinlar/Yayinlarimiz/2020/kuresel-salginda-yeni-normallesme-sureci-2020.pdf
• Zamanzadeh, V., Ghahramanian, A., Rassouli, M., Abbaszadeh, A., Alavi-Majd, H., & Nikanfar, A. R. (2015). Design and implementation content validity study: Development of an instrument for measuring patient-centered communication. Journal of caring sciences, 4(2), 165–178. https://doi.org/10.15171/jcs.2015.017
• Zenouzagh, Z. M. (2019). The effect of online summative and formative teacher assessment on teacher competences. Asia Pacific Education Review, 20(3), 343–359. doi:10.1007/s12564-018-9566-1
• Zheng, L., Chen, N.-S., Cui, P., & Zhang, X. (2019). A Systematic Review of Technology-Supported Peer Assessment Research: An Activity Theory Approach. International Review of Research in Open and Distributed Learning, 20(5), 168–191.