Measuring cognitive loads and attitudes of pre-service teachers in computer based testing environment

Abstract

This study aimed to reveal the attitudes of the students who were subjected to measurement and evaluation in an online testing environment towards the computer based testing (CBT) platform and the factors that affect their attitudes. It also examined the students’ cognitive loads in the exam designed with multimedia elements from message design principles by using the facilities of this platform. The study was conducted with a case study design. The participants consisted of 32 students. Participants took a previously used and redesigned English achievement test based on the multimedia design principle. Then, cognitive load scale and attitude towards computer scale were implemented. In addition, seven volunteer participants were interviewed. The results of the study showed that factors such as students’ equipment advantage, time, success, flexibility, individuality and less mental effort affected the participants’ attitudes towards the CBT environment. A significant difference was found between the scores of the participants in the cognitive load scale and the duration of the exam. There was a low and positive correlation between the cognitive load score of the participants for the listening test and the time to complete these questions, and a low and negative correlation between the correct answer scores of the participants in the listening test.

Keywords

• Cognitive load theory
• principles of message design
• online testing
• computer based testing (CBT)
• case study.

References

1. Akdemir, Ö. & Oğuz, A. (2008). Computer-based testing: an alternative for the assessment of Turkish undergraduate students, Computers and Education, 1, 1198-1204. https://doi.org/10.1016/j.compedu.2007.11.007
2. Aybek, E. C. (2012). Kâğıt-kalem formu ve bilgisayar ortamında uygulanan genel yetenek testinin psikometrik özelliklerinin karşılaştırılması [Master’s thesis, Ankara University]. YÖK Açıkbilim. https://acikbilim.yok.gov.tr/handle/20.500.12812/85211
3. Brünken, R., Seufert, T., & Paas, F. (2010). Measuring Cognitive Load. In J. L. Plass, R. Moreno, & R. Brünken (Eds.), Cognitive Load Theory (pp. 181–202). Cambridge University Press.
4. Bugbee, A.C. (1996). The Equivalence of Paper-and-Pencil and Computer-Based Testing. Journal of Research on Computing in Education, 28(3), 282. Retrieved September 19, 2023 from https://www.learntechlib.org/p/80961/.
5. Büyüköztürk, Ş., Kılıç Çakmak, E., Akgün, Ö. E., Karadeniz Oran, Ş., & Demirel, F., (2019). Bilimsel Araştırma Yöntemleri. Pegem Akademi.
6. Çaka, C. & Dursun, Ö.Ö. (2022). Evaluation of the effectiveness of different infographic designs. Journal of Educational Technology & Online Learning, 5(3), 519-534. https://doi.org/10.31681/jetol.1075352
7. Çetin, İ. (2016). Nitel içerik analizi. In Özden, M. Y. & Durdu, L. (Eds.), Eğitimde Üretim Tabanlı Çalışmalar İçin Nitel Araştırma Yöntemleri (pp. 125-148). Anı Yayıncılık.
8. Dinar, M., Kabakçı Yurdakul, I. & Dönmez, F. (2015). Measuring Cognitive Load in Test Items: Static Graphics versus Animated Graphics. Journal of Computer Assisted Learning, 31, 148-161. https://doi.org/10.1111/jcal.12086

9. Fraenkel, J. R., Wallen, N. E., ve Hyun, H. H. (2012). How to design and evaluate research in education (8th ed.). McGraw-Hill.
10. Hamad, W. (2023). Students’ experiences of learning in virtual classrooms facilitated by Google Classroom. Journal of Educational Technology and Online Learning, 6 (2), 362-383. http://doi.org/10.31681/jetol.1250095
11. Kirschner, P.A. (2003). Cognitive Load Theory: Implications of Cognitive Load Theory on the Design of Learning, Learning and Instruction, 12, 1-10. https://doi.org/10.1016/S0959-4752(01)00014-7
12. Kılıç, E. (2009). The effects of cognitive load in learning from goal based scenario designed multimedia learning environment for learners having different working memory capacities (Publication No. 255259) [Doctoral Dissertation, METU]. YÖK Tez, https://tez.yok.gov.tr/UlusalTezMerkezi/TezGoster?key=NtBAevXNhYaNqJFoAcdBdoczOCd_zbRfD8kVAKPsDubMuN-FCZVksYHFE-wH0v9e
13. Kılıç, E. ve Karadeniz, Ş. (2004). Hiper ortamlarda öğrencilerin bilişsel yüklenme ve kaybolma düzeylerinin belirlenmesi. Kuram ve Uygulamada Eğitim Yönetimi, 40, 56—579. Retrieved from https://dergipark.org.tr/tr/pub/kuey/issue/10357/126808
14. Lemmo, A., Mariotti, M-A. (2017, 1-5 February). From paper and pencil to computer-based assessment: some issues raised in the comparison, In. Dooley, T. & Gueudet, G. (Eds.), Proceedings of the Tenth Congress of the European Society for Research in Mathematics Education, DCU Institute of Education & ERME, 3540 – 3547.
15. Lindner, M. A., Eitel, A., Barenthien, J., & Köller, O. (2018). An integrative study on learning and testing with multimedia: Effects on students’ performance and metacognition. Learning and Instruction, 1(2), 1-9. https://doi.org/10.1016/j.learninstruc.2018.01.002
16. Mayer, R.E. (2003). The Promise of Multimedia Learning: Using The Same İnstructional Design Methods Across Different Media, Learning and Instruction, 13, 125-139. https://doi.org/10.1016/S0959-4752(02)00016-6
17. Özturan, T. (2016). The impact of computer assisted assessment on the exam success and attitudes of prospective English teachers. (Publication No. 435226) [ Master’s thesis, Hacettepe University]. YÖK Tez, https://tez.yok.gov.tr/UlusalTezMerkezi/TezGoster?key=cbOXH84ZayrLjc0tI-QXKtTWLLFqln0MiUoLfO-MxUyK83GNRlTkrzR5Bs7ix3I1
18. Paas, F. G., & Van Merriënboer, J. J. (1994). Instructional control of cognitive load in the training of complex cognitive tasks. Educational Psychology Review, 6, 351-371. https://doi.org/10.1007/BF02213420
19. Paas, F., Renkl, A. & Sweller, J. (2003). Cognitive load theory and instructional design: recent developments, Educational Psychologist, 38(1), 1-4. https://doi.org/10.1207/S15326985EP3801_1
20. Prisacari, A.A. & Danielson, J. (2017). Computer-based versus paper-based testing: Investigating testing mode with cognitive load and scratch paper use, Computers in Human Behavior, 77, 1-10. https://doi.org/10.1016/j.chb.2017.07.044
21. Şenel, S. (2018). Programlama Konusu “Kazanımları” ile “Sınıf İçi Ölçme Süreçlerinin” Bilişsel Düzeylerinin Karşılaştırılması. International Journal of Computers in Education, 1 (2), 1-20. Retrieved from https://dergipark.org.tr/en/pub/ijce/issue/42323/486158
22. Şenel, S., Pekdağ B., & Günaydın, S. (2018). Kimya öğretmenlerinin ölçme ve değerlendirme sürecinde yaşadıkları problemler ve yetersizlikler. Necatibey Eğitim Fakültesi Elektronik Fen ve Matematik Eğitimi Dergisi (EFMED). 12(1). 419-441. https://doi.org/10.17522/balikesirnef.437824
23. Sweller, J., van Merrienboer, J.J.G. & Paas FGWC. (1998). Cognitive Architecture and Instructional Design, Educational Psychology Review, 10(3), 251-296. https://doi.org/10.1023/A:1022193728205
24. Takır, A. (2011). The effect of an instruction designed by cognitive load theory principles on 7th grade students' achievement in algebra topics and cognitive load. (Publication No. 255259) [ Doctoral Dissertation, METU]. YÖK Tez, https://tez.yok.gov.tr/UlusalTezMerkezi/TezGoster?key=EEdeQgIdFRxX5NbvVau-AhjV-QzHr495E11DeqZt2L6gI88s6pwJxJAMh4_IhC9C
25. Woodside, A. G. (2010). Case Study Research: Theory, Methods, Practice. Emerald Group Pub. Ltd., ebrary Inc.
26. Yıldırım, A. & Şimşek, H. (2018). Sosyal Bilimlerde Nitel Araştırma Yöntemleri. Seçkin Yayıncılık.
27. Yıldırım, Z. (2016). Öğretim teknolojileri ve ileti tasarımı. In Çağıltay, K. & Göktaş, Y. (Eds), Öğretim Teknolojilerinin Temelleri: Teoriler, Araştırmalar, Eğilimler (279-296). Pegem Akademi.