Investigating Animation-Based Achievement Tests According to Various Variables

Year 2022, Volume: 9 Issue: 4, 33 - 52, 01.07.2022

Elif Güven Demir Yücel Öksüz

https://doi.org/10.17275/per.22.78.9.4

Abstract

This research aimed to investigate animation-based achievement tests according to the item format, psychometric features, students’ performance, and gender. The study sample consisted of 52 fifth-grade students in Samsun/Turkey in 2017-2018. Measures of the research were open-ended (OE), animation-based open-ended (AOE), multiple-choice (MC), and animation-based multiple-choice (AMC) achievement tests developed for the unit “Motions of the Earth” of the subject area “Earth and Universe” of the science course. Tests were compared to item difficulty, item discrimination, internal consistency levels, and students’ performance and gender. Additionally, predicting students’ science grades was tested by animation-based open-ended and multiple-choice tests. Paired sample t-tests, Pearson’s correlation coefficient, Friedman test, Wilcoxon test, two-way mixed ANOVA tests, and multiple regression analysis were applied to analyze research data. Research results show no significant difference between animation-based and traditional tests' validity and reliability levels. Research result also shows that students’ test scores were significantly higher at animation-based tests than traditional tests. Besides, students’ test scores differed according to the animation type used in tests. However, gender was not a significant variable on students' test scores. Finally, despite using animation as a significant predictor of Science grades, just animation based multiple-choice test has significantly predicted students’ Science grades. Future research can investigate variables that affect students' participation in animation-based tests and their opinions on animation-based tests. The finding regarding the animation type variable can be investigated in-depth in terms of the effect levels of the visual and auditory elements added to the animations by including an equal number of items on the test.

Keywords

animation, achievement, open-ended test, multiple-choice test, science

References

• Ally, B. A. (2012). Using pictures and words to understand recognition memory deterioration in amnestic mild cognitive impairment and Alzheimer’s disease: a review. Current Neurology and Neuroscience Reports, 12(6), 687–694. https://doi.org/10.1007/s11910-012-0310-7
• Alsawalmeh, Y. M., & Feldt, L. S. (1994). A modification of Feldt’s test of the equality of two independent alpha coefficients. Psychometrika, 59(1), 49–57. https://doi.org/10.1007/BF02294264
• Anıl, D., & Acar, M. (2008). Sınıf öğretmenlerinin ölçme değerlendirme sürecinde karşılaştıkları sorunlara ilişkin görüşleri. Journal of Yüzüncü Yıl University Faculty of Education, 5(2), 44-61. https://dergipark.org.tr/tr/pub/yyuefd/issue/13714/166032
• Arellano, M. D. C. (2013). Gender differences in reading comprehension achievement in English as a foreign language in Compulsory Secondary Education. Tejuelo: Didáctica de la Lengua y la Literatura. Educación, 17, 67-84.
• Atkinson, R. K. (2002). Optimizing learning from examples using animated pedagogical agents. Journal of Educational Psychology, 94(2), 416–427. https://doi.org/10.1037/0022-0663.94.2.416
• Atkinson, R., Mayer, R. E., & Merrill, M. M. (2005). Fostering social agency in multimedia learning: Examining the impact of an animated agent’s voice. Contemporary Educational Psychology, 30(1), 117-139. https://doi.org/10.1016/j.cedpsych.2004.07.001
• Barak, M., Ashkar, T., & Dori, Y. J. (2011). Learning Science via animated movies: Its effect on students’ thinking and motivation. Computers & Education, 56(3), 839–846. https://doi.org/10.1016/j.compedu.2010.10.025
• Bayat, N., Şekercioğlu, G., & Bakır, S. (2014). Okuduğunu anlama ve fen başarısı arasındaki ilişkinin belirlenmesi. Education and Science, 39(176). http://dx.doi.org/10.15390/EB.2014.3693
• Birgili, B. (2014). Open ended questions as an alternative to multiple choice: dilemma in Turkish examination system [Master dissertation, Middle East Technical University]. OpenMETU. https://hdl.handle.net/11511/23866
• Braun, H. I., Bennett, R. E., Frye, D., & Soloway, E. (1990). Scoring constructed responses using expert systems. Journal of Educational Measurement, 27(2), 93-108. https://doi.org/10.1111/j.1745-3984.1990.tb00736.x
• Burke, S. C., Snyder, S., & Rager, R. C. (2009). An assessment of faculty usage of YouTube as a teaching resource. Internet Journal of Allied Health Sciences and Practice, 7(1), 8. https://nsuworks.nova.edu/ijahsp/vol7/iss1/8/
• Cachia, R., Ferrari, A., Ala-Mutka, K., & Punie, Y. (2010). Creative learning and innovative teaching: Final report on the study on creativity and innovation in education in the EU member states. JRC European Commission. https://publications.jrc.ec.europa.eu/repository/bitstream/JRC62370/jrc62370.pdf
• Chan, C. K. Y. (2015). Use of animation in engaging teachers and students in assessment in Hong Kong higher education. Innovations in Education and Teaching International, 52(5), 474-484. https://doi.org/10.1080/14703297.2013.847795
• Chao, T., Chen, J., Star, J. R., & Dede, C. (2016). Using digital resources for motivation and engagement in learning mathematics: reflections from teachers and students. Digital Experiences in Mathematics Education, 2(3), 253-277. https://doi.org/10.1007/s40751-016-0024-6
• Çi̇ftçi̇, Ö., & Temi̇zyürek, F. (2008). İlköğretim 5. sınıf öğrencilerinin okuduğunu anlama becerilerinin ölçülmesi. Mustafa Kemal University Journal of Social Sciences Institue, 5(9), 109-129. https://dergipark.org.tr/tr/pub/mkusbed/issue/19561/208532
• Dalacosta, K., Kamariotaki-Paparrigopoulou, M., Palyvos, J. A., & Spyrellis, N. (2009). Multimedia application with animated cartoons for teaching Science in elementary education. Computers & Education, 52(4), 741–748. https://doi.org/10.1016/j.compedu.2008.11.018
• Dancy, M.H. (2000). Investigating animations for assessment with an animated version of the Force Concept Inventory [Doctoral dissertation, North Carolina State University, Raleigh]. https://www.learntechlib.org/p/128748/
• Dancy, M. H. & Beichner, R. (2006). Impact of animation on assessment of conceptual understanding in physics. Physical Review Special Topics - Physics Education Research, 2. https://doi.org/10.1103/PhysRevSTPER.2.010104
• Daşdemi̇r, İ., & Doymuş, K. (2016). Fen ve teknoloji dersinde animasyon kullanımının öğrencilerin akademik başarılarına, öğrenilen bilgilerin kalıcılığına ve bilimsel süreç becerilerine etkisi. Pegem Education and Instruction Journal, 2(3), 33-42. https://dergipark.org.tr/tr/pub/befdergi/issue/23147/247268
• Denning, D. (1992). Video in theory and practice: Issues for classroom use and teacher video evaluation. Innature Productions.https://www.academia.edu/4666293/Video_in_Theory_and_Practice_Issues_for_Classroom_Use_and_Teacher_Video_Evaluation
• Dewan, P. (2015). Words versus pictures: leveraging the research on visual communication. Partnership: The Canadian Journal of Library and Information Practice and Research, 10(1). https://doi.org/10.21083/partnership.v10i1.3137
• Dindar, M., Kabakçı Yurdakul, I., & İnan Dönmez, F. (2015). Measuring cognitive load in test items: static graphics versus animated graphics. Journal of Computer Assisted Learning, 31(2), 148-161. https://doi.org/10.1111/jcal.12086
• Doğan, S. (2012). Kavram haritası ve yapılandırılmış grid tekniğinin çoktan seçmeli testlerle karşılaştırılması (Publication No. 319636) [Master dissertation, Mersin University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
• Ebel, R. L. & Frisbie, D. A. (1991). Essentials of educational measurement (5th ed.). Prentice-Hall of India. https://ebookppsunp.files.wordpress.com/2016/06/robert_l-ebel_david_a _frisbie_essentials_of_edbookfi-org.pdf
• English, J. (2020). Animation in online school science assessment: the validation of assessment for learning and individual development program. In: Unsworth L. (eds) Learning from animations in science education. innovations in science education and technology, vol 25. Springer, Cham. https://doi.org/10.1007/978-3-030-56047-8_11
• Ercan, O., Bilen, K., & Bulut, A. (2014). The effect of web-based instruction with educational animation content at sensory organs subject on students’ academic achievement and attitudes. Procedia - Social and Behavioral Sciences, 116, 2430-2436. https://doi.org/10.1016/j.sbspro.2014.01.587
• Ferrari, A., Cachia, R., & Punie, Y. (2009). Innovation and creativity in education and training in the EU member states: Fostering creative learning and supporting innovative teaching. JRC European Commission. http://ftp.jrc.es/EURdoc/JRC52374_TN.pdf
• Field, Andy (2013) Discovering statistics using IBM SPSS Statistics: and sex and drugs and rock ‘n’ roll (4th ed.). Sage.
• Güler, N. (2017), Eğitimde ölçme ve değerlendirme (10th ed.), Pegem Akademi Publishing
• Gültekin, C. (2014). Ortaöğretim öğrencileri ile üniversite öğrencilerinin hal değişimi, çözeltiler ve çözünürlük konuları ile ilgili grafik çizme okuma ve yorumlama becerilerinin karşılaştırılması [Doctoral dissertation, University of Balıkesir]. Dspace. https://hdl.handle.net/20.500.12462/2836
• Güven Demir, E. (2018). Ters yüz sınıf modeline dayalı uygulamaların ilkokul 4. Sınıf öğrencilerinin akademik aşarı ve planlama becerilerine etkisi (Publication No.519317) [Doctoral dissertation, University of Ondokuz Mayıs]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
• Hair, J., Black, W. C., Babin, B. J. and Anderson, R. E. (2010), Multivariate data analysis (7th ed.). Pearson Education International.
• Hwang, I., Tam, M., Lam, S. L., & Lam, P. (2012). Review of use of animation as a supplementary learning material of physiology content in four academic years. Electronic Journal of e-Learning, 10 (4), 368–377. http://www.ejel.org/volume10/issue4
• Kan, A. ve Kayapınar, U. (2006). Yabancı dil eğitiminde aynı davranışları yoklayan çoktan seçmeli ve kısa cevaplı iki testin madde ve test özelliklerinin karşılaştırılması. Education and Science, 32(142). http://egitimvebilim.ted.org.tr/index.php/EB/article/view/844
• Karamustafaoğlu, S. ve Tutar, M. (2016). 6. sınıf dünya’mız, ay ve yaşam kaynağımız güneş ünitesi’ne yönelik bir başarı testi geliştirme. In Ö. Demirel, S. Dinçer (Eds.), Eğitim bilimlerinde yenilikler ve nitelik arayışı (2nd ed., pp.303–320), Pegem Atıf İndeksi, 303–320. http://dx.doi.org/10.14527/9786053183563b2.020
• Kayaoglu, M. N., Akbas, R. D., & Öztürk, Z. (2011). A small scale experimental study: Using animations to learn vocabulary. TOJET: The Turkish Online Journal of Educational Technology, 10 (2). https://files.eric.ed.gov/fulltext/EJ932222.pdf
• King, J. A. (2015). getting the picture: a cartoon-based assessment tool for complex trauma in children [Doctoral dissertation, University of Pennsylvania].https://repository.upenn.edu/edissertations_sp2/68/
• Klufa, J. (2015). Multiple choice question tests–advantages and disadvantages. In K. Psarris (Ed.), Proceedings of the 2015 international conference on education and modern educational technologies (pp. 39-42). http://www.inase.org/library/2015/
• Koumi, J. (2006). Designing video and multimedia for open and flexible learning. Routledge. https://doi.org/10.4324/9780203966280
• Kutlu, Ö., Yıldırım, Ö., Bilican, S., & Kumandaş, H. (2011). İlköğretim 5. sınıf öğrencilerinin okuduğunu anlamada başarılı olup-olmama durumlarının kestirilmesinde etkili olan değişkenlerin incelenmesi. Journal of Measurement and Evaluation in Education and Psychology, 2(1), 132-139. https://dergipark.org.tr/tr/pub/epod/issue/5806/77235
• Lester, J., Callaway, C.B., Stone, B., & Towns, S.G. (1997). Mixed initiative problem solving with animated pedagogical agents. AAAI Technical Report. https://www.aaai.org/Library/Symposia/Spring/1997/ss97-04-018.php
• Liu, C., & Elms, P. (2019). Animating student engagement: The impacts of cartoon instructional videos on learning experience. Research in Learning Technology, 27. https://doi.org/10.25304/rlt.v27.2124
• Malone, S., & Brünken, R. (2013). Assessment of driving expertise using multiple choice questions including static vs. animated presentation of driving scenarios. Accident Analysis & Prevention, 51, 112-119. https://doi.org/10.1016/j.aap.2012.11.003
• Martins, I. P. & Veiga, L. (2001). Early science education: exploring familiar contexts to improve the understanding of some basic scientific concepts, European Early Childhood Education Research Journal, 9:2, 69-82. https://doi.org/10.1080/13502930185208771
• Mayer, R. E. (2005). Cognitive theory of multimedia learning: The cambridge handbook of multimedia learning (R. E. Mayer, Ed.). Cambridge University Press. 31–48. https://doi.org/10.1017/CBO9780511816819.004
• Mayer, R. E. (2009). Multimedia learning (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511811678
• Mayer, R. E. (2014). Incorporating motivation into multimedia learning. Learning and Instruction, 29, 171-173. https://doi.org/10.1016/j.learninstruc.2013.04.003
• Moreno, R., Mayer, R. E., Spires, H. A., & Lester, J. C. (2001). The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction, 19(2), 177–213.https://doi.org/10.1207/S1532690XCI1902_02
• Moussiades, L., Kazanidis, I., & Iliopoulou, A. (2019). A framework for the development of educational video: An empirical approach. Innovations in Education and Teaching International, 56(2), 217-228. https://doi.org/10.1080/14703297.2017.1399809
• Nass, C. I., & Brave, S. (2005). Wired for speech: How voice activates and advances the human-computer relationship. MIT Press.
• Öksüz, Y., & Demir, E. G. (2019). Açık uçlu ve çoktan seçmeli başarı testlerinin psikometrik özellikleri ve öğrenci performansı açısından karşılaştırılması, Hacettepe University Journal of Education, 34(1), 259-282. https://doi.org/10.16986/HUJE.2018040550
• Qiu, S., Xia, Y., Tian, F. et al. (2020). Using a cartoon questionnaire to improve consent process in children: a randomized controlled survey. Pediatric Research. https://doi.org/10.1038/s41390-020-01227-2
• Quaigrain, K., & Arhin, A. K. (2017). Using reliability and item analysis to evaluate a teacher-developed test in educational measurement and evaluation. Cogent Education, 4(1), 1301013. https://doi.org/10.1080/2331186X.2017.1301013
• Sarıgül, Z. (2009). Çoktan seçmeli, yapılandırılmış grid, ve kavram haritası tekniklerinin öğrenci başarısını ölçme açısından etkililiğinin incelenmesi ve öğrencilerin bu tekniklere ilişkin görüşleri (Publication No. 241764) [Master dissertation, University of Abant İzzet Baysal]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
• Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8(4), 350–353. https://doi.org/10.1037/1040-3590.8.4.350
• Schroeder, N. L., Adesope, O. O., & Gilbert, R. B. (2013). How effective are pedagogical agents for learning? A meta-analytic review. Journal of Educational Computing Research, 49(1), 1-39. https://doi.org/10.2190/EC.49.1.a
• Schuwirth, L. W. T., van der Vleuten, C. P. M., & Donkers, H. H. L. M. (1996). A closer look at cueing effects in multiple‐choice questions. Medical Education, 30(1), 44-49. https://doi.org/10.1111/j.1365-2923.1996.tb00716.x
• Schwartz, D. L., & Hartman, K. (2007). It is not television anymore: Designing digital video for learning and assessment (R. Goldman, R. Pea, B. Barron, S. J. Derry, Eds.). Video research in the learning sciences, 335-348. https://doi.org/10.4324/9780203877258
• Stith B. J. (2004). Use of animation in teaching cell biology. Cell Biology Education, 3(3), 181–188. https://doi.org/10.1187/cbe.03-10-0018
• Struyven, K., Dochy, F., & Janssens, S. (2005). Students’ perceptions about evaluation and assessment in higher education: a review 1. Assessment & Evaluation in Higher Education, 30(4), 325-341. https://doi.org/10.1080/02602930500099102
• Strømme, T.A., Mork, S.M. (2020). Students’ conceptual sense-making of animations and static visualizations of protein synthesis: a sociocultural hypothesis explaining why animations may be beneficial for student learning. Research in Science Education. https://doi.org/10.1007/s11165-020-09920-2
• van der Meij, H., van der Meij, J., & Harmsen, R. (2015). Animated pedagogical agents effects on enhancing student motivation and learning in a science inquiry learning environment. Educational Technology Research and Development, 63(3), 381-403. https://doi.org/10.1007/s11423-015-9378-5
• van Griethuijsen, R.A.L.F., van Eijck, M.W., Haste, H., den Brok, P. J., Skinner, N. C., Mansour, N., Savran Gencer, A., Boujaoude, S. (2015). Global patterns in students’ views of Science and ınterest in Science, Research in Science Education, 45, 581–603. https://doi.org/10.1007/s11165-014-9438-6
• Wouters, P., Paas, F., & van Merriënboer, J. J. (2008). How to optimize learning from animated models: A review of guidelines based on cognitive load. Review of Educational Research, 78(3), 645-675.https://doi.org/10.3102/0034654308320320
• Wu, H.-C., Chang, C.-Y., Chen, C.-L. D., Yeh, T.-K., & Liu, C.-C. (2010). Comparison of earth science achievement between animation-based and graphic-based testing designs. Research in Science Education, 40(5), 639-673. https://doi.org/10.1007/s11165-009-9138-9
• Wu, H. C., Yeh, T. K., & Chang, C. Y. (2010). The design of an animation-based test system in the area of Earth sciences. British Journal of Educational Technology, 41(3), 53-57. https://doi.org/10.1111/j.1467-8535.2009.00977.x
• Yezierski, E. J., & Birk, J. P. (2006). Misconceptions about the particulate nature of matter. Using animations to close the gender gap. Journal of Chemical Education, 83, 954-960. https://doi.org/10.1021/ed083p954