Investigating Animation-Based Achievement Tests According to Various Variables

Elif Güven Demir; Yücel Öksüz

doi:10.17275/per.22.78.9.4

EN

Investigating Animation-Based Achievement Tests According to Various Variables

Abstract

This research aimed to investigate animation-based achievement tests according to the item format, psychometric features, students’ performance, and gender. The study sample consisted of 52 fifth-grade students in Samsun/Turkey in 2017-2018. Measures of the research were open-ended (OE), animation-based open-ended (AOE), multiple-choice (MC), and animation-based multiple-choice (AMC) achievement tests developed for the unit “Motions of the Earth” of the subject area “Earth and Universe” of the science course. Tests were compared to item difficulty, item discrimination, internal consistency levels, and students’ performance and gender. Additionally, predicting students’ science grades was tested by animation-based open-ended and multiple-choice tests. Paired sample t-tests, Pearson’s correlation coefficient, Friedman test, Wilcoxon test, two-way mixed ANOVA tests, and multiple regression analysis were applied to analyze research data. Research results show no significant difference between animation-based and traditional tests' validity and reliability levels. Research result also shows that students’ test scores were significantly higher at animation-based tests than traditional tests. Besides, students’ test scores differed according to the animation type used in tests. However, gender was not a significant variable on students' test scores. Finally, despite using animation as a significant predictor of Science grades, just animation based multiple-choice test has significantly predicted students’ Science grades. Future research can investigate variables that affect students' participation in animation-based tests and their opinions on animation-based tests. The finding regarding the animation type variable can be investigated in-depth in terms of the effect levels of the visual and auditory elements added to the animations by including an equal number of items on the test.

Keywords

animation, achievement, open-ended test, multiple-choice test, science

References

Ally, B. A. (2012). Using pictures and words to understand recognition memory deterioration in amnestic mild cognitive impairment and Alzheimer’s disease: a review. Current Neurology and Neuroscience Reports, 12(6), 687–694. https://doi.org/10.1007/s11910-012-0310-7
Alsawalmeh, Y. M., & Feldt, L. S. (1994). A modification of Feldt’s test of the equality of two independent alpha coefficients. Psychometrika, 59(1), 49–57. https://doi.org/10.1007/BF02294264
Anıl, D., & Acar, M. (2008). Sınıf öğretmenlerinin ölçme değerlendirme sürecinde karşılaştıkları sorunlara ilişkin görüşleri. Journal of Yüzüncü Yıl University Faculty of Education, 5(2), 44-61. https://dergipark.org.tr/tr/pub/yyuefd/issue/13714/166032
Arellano, M. D. C. (2013). Gender differences in reading comprehension achievement in English as a foreign language in Compulsory Secondary Education. Tejuelo: Didáctica de la Lengua y la Literatura. Educación, 17, 67-84.
Atkinson, R. K. (2002). Optimizing learning from examples using animated pedagogical agents. Journal of Educational Psychology, 94(2), 416–427. https://doi.org/10.1037/0022-0663.94.2.416
Atkinson, R., Mayer, R. E., & Merrill, M. M. (2005). Fostering social agency in multimedia learning: Examining the impact of an animated agent’s voice. Contemporary Educational Psychology, 30(1), 117-139. https://doi.org/10.1016/j.cedpsych.2004.07.001
Barak, M., Ashkar, T., & Dori, Y. J. (2011). Learning Science via animated movies: Its effect on students’ thinking and motivation. Computers & Education, 56(3), 839–846. https://doi.org/10.1016/j.compedu.2010.10.025
Bayat, N., Şekercioğlu, G., & Bakır, S. (2014). Okuduğunu anlama ve fen başarısı arasındaki ilişkinin belirlenmesi. Education and Science, 39(176). http://dx.doi.org/10.15390/EB.2014.3693
Birgili, B. (2014). Open ended questions as an alternative to multiple choice: dilemma in Turkish examination system [Master dissertation, Middle East Technical University]. OpenMETU. https://hdl.handle.net/11511/23866
Braun, H. I., Bennett, R. E., Frye, D., & Soloway, E. (1990). Scoring constructed responses using expert systems. Journal of Educational Measurement, 27(2), 93-108. https://doi.org/10.1111/j.1745-3984.1990.tb00736.x

Burke, S. C., Snyder, S., & Rager, R. C. (2009). An assessment of faculty usage of YouTube as a teaching resource. Internet Journal of Allied Health Sciences and Practice, 7(1), 8. https://nsuworks.nova.edu/ijahsp/vol7/iss1/8/
Cachia, R., Ferrari, A., Ala-Mutka, K., & Punie, Y. (2010). Creative learning and innovative teaching: Final report on the study on creativity and innovation in education in the EU member states. JRC European Commission. https://publications.jrc.ec.europa.eu/repository/bitstream/JRC62370/jrc62370.pdf
Chan, C. K. Y. (2015). Use of animation in engaging teachers and students in assessment in Hong Kong higher education. Innovations in Education and Teaching International, 52(5), 474-484. https://doi.org/10.1080/14703297.2013.847795
Chao, T., Chen, J., Star, J. R., & Dede, C. (2016). Using digital resources for motivation and engagement in learning mathematics: reflections from teachers and students. Digital Experiences in Mathematics Education, 2(3), 253-277. https://doi.org/10.1007/s40751-016-0024-6
Çi̇ftçi̇, Ö., & Temi̇zyürek, F. (2008). İlköğretim 5. sınıf öğrencilerinin okuduğunu anlama becerilerinin ölçülmesi. Mustafa Kemal University Journal of Social Sciences Institue, 5(9), 109-129. https://dergipark.org.tr/tr/pub/mkusbed/issue/19561/208532
Dalacosta, K., Kamariotaki-Paparrigopoulou, M., Palyvos, J. A., & Spyrellis, N. (2009). Multimedia application with animated cartoons for teaching Science in elementary education. Computers & Education, 52(4), 741–748. https://doi.org/10.1016/j.compedu.2008.11.018
Dancy, M.H. (2000). Investigating animations for assessment with an animated version of the Force Concept Inventory [Doctoral dissertation, North Carolina State University, Raleigh]. https://www.learntechlib.org/p/128748/
Dancy, M. H. & Beichner, R. (2006). Impact of animation on assessment of conceptual understanding in physics. Physical Review Special Topics - Physics Education Research, 2. https://doi.org/10.1103/PhysRevSTPER.2.010104
Daşdemi̇r, İ., & Doymuş, K. (2016). Fen ve teknoloji dersinde animasyon kullanımının öğrencilerin akademik başarılarına, öğrenilen bilgilerin kalıcılığına ve bilimsel süreç becerilerine etkisi. Pegem Education and Instruction Journal, 2(3), 33-42. https://dergipark.org.tr/tr/pub/befdergi/issue/23147/247268
Denning, D. (1992). Video in theory and practice: Issues for classroom use and teacher video evaluation. Innature Productions.https://www.academia.edu/4666293/Video_in_Theory_and_Practice_Issues_for_Classroom_Use_and_Teacher_Video_Evaluation
Dewan, P. (2015). Words versus pictures: leveraging the research on visual communication. Partnership: The Canadian Journal of Library and Information Practice and Research, 10(1). https://doi.org/10.21083/partnership.v10i1.3137
Dindar, M., Kabakçı Yurdakul, I., & İnan Dönmez, F. (2015). Measuring cognitive load in test items: static graphics versus animated graphics. Journal of Computer Assisted Learning, 31(2), 148-161. https://doi.org/10.1111/jcal.12086
Doğan, S. (2012). Kavram haritası ve yapılandırılmış grid tekniğinin çoktan seçmeli testlerle karşılaştırılması (Publication No. 319636) [Master dissertation, Mersin University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
Ebel, R. L. & Frisbie, D. A. (1991). Essentials of educational measurement (5th ed.). Prentice-Hall of India. https://ebookppsunp.files.wordpress.com/2016/06/robert_l-ebel_david_a _frisbie_essentials_of_edbookfi-org.pdf
English, J. (2020). Animation in online school science assessment: the validation of assessment for learning and individual development program. In: Unsworth L. (eds) Learning from animations in science education. innovations in science education and technology, vol 25. Springer, Cham. https://doi.org/10.1007/978-3-030-56047-8_11
Ercan, O., Bilen, K., & Bulut, A. (2014). The effect of web-based instruction with educational animation content at sensory organs subject on students’ academic achievement and attitudes. Procedia - Social and Behavioral Sciences, 116, 2430-2436. https://doi.org/10.1016/j.sbspro.2014.01.587
Ferrari, A., Cachia, R., & Punie, Y. (2009). Innovation and creativity in education and training in the EU member states: Fostering creative learning and supporting innovative teaching. JRC European Commission. http://ftp.jrc.es/EURdoc/JRC52374_TN.pdf
Field, Andy (2013) Discovering statistics using IBM SPSS Statistics: and sex and drugs and rock ‘n’ roll (4th ed.). Sage.
Güler, N. (2017), Eğitimde ölçme ve değerlendirme (10th ed.), Pegem Akademi Publishing
Gültekin, C. (2014). Ortaöğretim öğrencileri ile üniversite öğrencilerinin hal değişimi, çözeltiler ve çözünürlük konuları ile ilgili grafik çizme okuma ve yorumlama becerilerinin karşılaştırılması [Doctoral dissertation, University of Balıkesir]. Dspace. https://hdl.handle.net/20.500.12462/2836
Güven Demir, E. (2018). Ters yüz sınıf modeline dayalı uygulamaların ilkokul 4. Sınıf öğrencilerinin akademik aşarı ve planlama becerilerine etkisi (Publication No.519317) [Doctoral dissertation, University of Ondokuz Mayıs]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
Hair, J., Black, W. C., Babin, B. J. and Anderson, R. E. (2010), Multivariate data analysis (7th ed.). Pearson Education International.
Hwang, I., Tam, M., Lam, S. L., & Lam, P. (2012). Review of use of animation as a supplementary learning material of physiology content in four academic years. Electronic Journal of e-Learning, 10 (4), 368–377. http://www.ejel.org/volume10/issue4
Kan, A. ve Kayapınar, U. (2006). Yabancı dil eğitiminde aynı davranışları yoklayan çoktan seçmeli ve kısa cevaplı iki testin madde ve test özelliklerinin karşılaştırılması. Education and Science, 32(142). http://egitimvebilim.ted.org.tr/index.php/EB/article/view/844
Karamustafaoğlu, S. ve Tutar, M. (2016). 6. sınıf dünya’mız, ay ve yaşam kaynağımız güneş ünitesi’ne yönelik bir başarı testi geliştirme. In Ö. Demirel, S. Dinçer (Eds.), Eğitim bilimlerinde yenilikler ve nitelik arayışı (2nd ed., pp.303–320), Pegem Atıf İndeksi, 303–320. http://dx.doi.org/10.14527/9786053183563b2.020
Kayaoglu, M. N., Akbas, R. D., & Öztürk, Z. (2011). A small scale experimental study: Using animations to learn vocabulary. TOJET: The Turkish Online Journal of Educational Technology, 10 (2). https://files.eric.ed.gov/fulltext/EJ932222.pdf
King, J. A. (2015). getting the picture: a cartoon-based assessment tool for complex trauma in children [Doctoral dissertation, University of Pennsylvania].https://repository.upenn.edu/edissertations_sp2/68/
Klufa, J. (2015). Multiple choice question tests–advantages and disadvantages. In K. Psarris (Ed.), Proceedings of the 2015 international conference on education and modern educational technologies (pp. 39-42). http://www.inase.org/library/2015/
Koumi, J. (2006). Designing video and multimedia for open and flexible learning. Routledge. https://doi.org/10.4324/9780203966280
Kutlu, Ö., Yıldırım, Ö., Bilican, S., & Kumandaş, H. (2011). İlköğretim 5. sınıf öğrencilerinin okuduğunu anlamada başarılı olup-olmama durumlarının kestirilmesinde etkili olan değişkenlerin incelenmesi. Journal of Measurement and Evaluation in Education and Psychology, 2(1), 132-139. https://dergipark.org.tr/tr/pub/epod/issue/5806/77235
Lester, J., Callaway, C.B., Stone, B., & Towns, S.G. (1997). Mixed initiative problem solving with animated pedagogical agents. AAAI Technical Report. https://www.aaai.org/Library/Symposia/Spring/1997/ss97-04-018.php
Liu, C., & Elms, P. (2019). Animating student engagement: The impacts of cartoon instructional videos on learning experience. Research in Learning Technology, 27. https://doi.org/10.25304/rlt.v27.2124
Malone, S., & Brünken, R. (2013). Assessment of driving expertise using multiple choice questions including static vs. animated presentation of driving scenarios. Accident Analysis & Prevention, 51, 112-119. https://doi.org/10.1016/j.aap.2012.11.003
Martins, I. P. & Veiga, L. (2001). Early science education: exploring familiar contexts to improve the understanding of some basic scientific concepts, European Early Childhood Education Research Journal, 9:2, 69-82. https://doi.org/10.1080/13502930185208771
Mayer, R. E. (2005). Cognitive theory of multimedia learning: The cambridge handbook of multimedia learning (R. E. Mayer, Ed.). Cambridge University Press. 31–48. https://doi.org/10.1017/CBO9780511816819.004
Mayer, R. E. (2009). Multimedia learning (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511811678
Mayer, R. E. (2014). Incorporating motivation into multimedia learning. Learning and Instruction, 29, 171-173. https://doi.org/10.1016/j.learninstruc.2013.04.003
Moreno, R., Mayer, R. E., Spires, H. A., & Lester, J. C. (2001). The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction, 19(2), 177–213.https://doi.org/10.1207/S1532690XCI1902_02
Moussiades, L., Kazanidis, I., & Iliopoulou, A. (2019). A framework for the development of educational video: An empirical approach. Innovations in Education and Teaching International, 56(2), 217-228. https://doi.org/10.1080/14703297.2017.1399809
Nass, C. I., & Brave, S. (2005). Wired for speech: How voice activates and advances the human-computer relationship. MIT Press.
Öksüz, Y., & Demir, E. G. (2019). Açık uçlu ve çoktan seçmeli başarı testlerinin psikometrik özellikleri ve öğrenci performansı açısından karşılaştırılması, Hacettepe University Journal of Education, 34(1), 259-282. https://doi.org/10.16986/HUJE.2018040550
Qiu, S., Xia, Y., Tian, F. et al. (2020). Using a cartoon questionnaire to improve consent process in children: a randomized controlled survey. Pediatric Research. https://doi.org/10.1038/s41390-020-01227-2
Quaigrain, K., & Arhin, A. K. (2017). Using reliability and item analysis to evaluate a teacher-developed test in educational measurement and evaluation. Cogent Education, 4(1), 1301013. https://doi.org/10.1080/2331186X.2017.1301013
Sarıgül, Z. (2009). Çoktan seçmeli, yapılandırılmış grid, ve kavram haritası tekniklerinin öğrenci başarısını ölçme açısından etkililiğinin incelenmesi ve öğrencilerin bu tekniklere ilişkin görüşleri (Publication No. 241764) [Master dissertation, University of Abant İzzet Baysal]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezSorguSonucYeni.jsp
Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8(4), 350–353. https://doi.org/10.1037/1040-3590.8.4.350
Schroeder, N. L., Adesope, O. O., & Gilbert, R. B. (2013). How effective are pedagogical agents for learning? A meta-analytic review. Journal of Educational Computing Research, 49(1), 1-39. https://doi.org/10.2190/EC.49.1.a
Schuwirth, L. W. T., van der Vleuten, C. P. M., & Donkers, H. H. L. M. (1996). A closer look at cueing effects in multiple‐choice questions. Medical Education, 30(1), 44-49. https://doi.org/10.1111/j.1365-2923.1996.tb00716.x
Schwartz, D. L., & Hartman, K. (2007). It is not television anymore: Designing digital video for learning and assessment (R. Goldman, R. Pea, B. Barron, S. J. Derry, Eds.). Video research in the learning sciences, 335-348. https://doi.org/10.4324/9780203877258
Stith B. J. (2004). Use of animation in teaching cell biology. Cell Biology Education, 3(3), 181–188. https://doi.org/10.1187/cbe.03-10-0018
Struyven, K., Dochy, F., & Janssens, S. (2005). Students’ perceptions about evaluation and assessment in higher education: a review 1. Assessment & Evaluation in Higher Education, 30(4), 325-341. https://doi.org/10.1080/02602930500099102
Strømme, T.A., Mork, S.M. (2020). Students’ conceptual sense-making of animations and static visualizations of protein synthesis: a sociocultural hypothesis explaining why animations may be beneficial for student learning. Research in Science Education. https://doi.org/10.1007/s11165-020-09920-2
van der Meij, H., van der Meij, J., & Harmsen, R. (2015). Animated pedagogical agents effects on enhancing student motivation and learning in a science inquiry learning environment. Educational Technology Research and Development, 63(3), 381-403. https://doi.org/10.1007/s11423-015-9378-5
van Griethuijsen, R.A.L.F., van Eijck, M.W., Haste, H., den Brok, P. J., Skinner, N. C., Mansour, N., Savran Gencer, A., Boujaoude, S. (2015). Global patterns in students’ views of Science and ınterest in Science, Research in Science Education, 45, 581–603. https://doi.org/10.1007/s11165-014-9438-6
Wouters, P., Paas, F., & van Merriënboer, J. J. (2008). How to optimize learning from animated models: A review of guidelines based on cognitive load. Review of Educational Research, 78(3), 645-675.https://doi.org/10.3102/0034654308320320
Wu, H.-C., Chang, C.-Y., Chen, C.-L. D., Yeh, T.-K., & Liu, C.-C. (2010). Comparison of earth science achievement between animation-based and graphic-based testing designs. Research in Science Education, 40(5), 639-673. https://doi.org/10.1007/s11165-009-9138-9
Wu, H. C., Yeh, T. K., & Chang, C. Y. (2010). The design of an animation-based test system in the area of Earth sciences. British Journal of Educational Technology, 41(3), 53-57. https://doi.org/10.1111/j.1467-8535.2009.00977.x
Yezierski, E. J., & Birk, J. P. (2006). Misconceptions about the particulate nature of matter. Using animations to close the gender gap. Journal of Chemical Education, 83, 954-960. https://doi.org/10.1021/ed083p954

Details

Primary Language

English

Subjects

Other Fields of Education

Journal Section

Research Article

Authors

Elif Güven Demir ^*
0000-0001-6685-5341
Türkiye

Yücel Öksüz
0000-0002-9310-7506
Türkiye

Publication Date

July 1, 2022

Submission Date

April 24, 2021

Acceptance Date

January 10, 2022

Published in Issue

Year 2022 Volume: 9 Number: 4

DOI

https://doi.org/10.17275/per.22.78.9.4

IZ

https://izlik.org/JA25GA83FX

APA

Güven Demir, E., & Öksüz, Y. (2022). Investigating Animation-Based Achievement Tests According to Various Variables. Participatory Educational Research, 9(4), 33-52. https://doi.org/10.17275/per.22.78.9.4

AMA

1.Güven Demir E, Öksüz Y. Investigating Animation-Based Achievement Tests According to Various Variables. PER. 2022;9(4):33-52. doi:10.17275/per.22.78.9.4

Chicago

Güven Demir, Elif, and Yücel Öksüz. 2022. “Investigating Animation-Based Achievement Tests According to Various Variables”. Participatory Educational Research 9 (4): 33-52. https://doi.org/10.17275/per.22.78.9.4.

EndNote

Güven Demir E, Öksüz Y (July 1, 2022) Investigating Animation-Based Achievement Tests According to Various Variables. Participatory Educational Research 9 4 33–52.

IEEE

[1]E. Güven Demir and Y. Öksüz, “Investigating Animation-Based Achievement Tests According to Various Variables”, PER, vol. 9, no. 4, pp. 33–52, July 2022, doi: 10.17275/per.22.78.9.4.

ISNAD

Güven Demir, Elif - Öksüz, Yücel. “Investigating Animation-Based Achievement Tests According to Various Variables”. Participatory Educational Research 9/4 (July 1, 2022): 33-52. https://doi.org/10.17275/per.22.78.9.4.

JAMA

1.Güven Demir E, Öksüz Y. Investigating Animation-Based Achievement Tests According to Various Variables. PER. 2022;9:33–52.

MLA

Güven Demir, Elif, and Yücel Öksüz. “Investigating Animation-Based Achievement Tests According to Various Variables”. Participatory Educational Research, vol. 9, no. 4, July 2022, pp. 33-52, doi:10.17275/per.22.78.9.4.

Vancouver

1.Elif Güven Demir, Yücel Öksüz. Investigating Animation-Based Achievement Tests According to Various Variables. PER. 2022 Jul. 1;9(4):33-52. doi:10.17275/per.22.78.9.4