Development, Reliability, and Validity of Open-ended Test to Measure Student’s Digital Literacy Skill

Year 2019, , 504 - 516, 01.10.2019

Riki Perdana , Riwayani Riwayani Jumadi Jumadi Dadan Rosana

https://doi.org/10.24331/ijere.628309

Cited By: 6

Abstract

Thisstudy aims to developing open-ended test for measuring digital literacy skills of students. There are three step of this study, it was defining the construc and formatting objectives, validity by expert review and item administration. The open-ended test was develop based on five components of digital literacy skill: information, communication, content creation, safety, and problem-solving. The open-ended test was initially piloted on four group of samples: interviews and validity by physics expert (2 Professor, 1 Doctoral), professional teachers (N=2), and college students of graduate school (N=2). Modification were made afterwards and the test was administered to a group of science students from high school in Yogyakarta province, Indonesia (N=129). Result of this study are: (1) The open-ended test instrumen proved valid and worthy of use to measure digital literacy skill; (2) Based on expert validation, the teset is valid with CVR value 1.00; (3) the average value and standard deviation of INFIT MNSQ is .83-1.21 (fit with rasch model/good item); (4) The item has a good degree of difficulty with range of difficulty level between (moderate level). The student’s digital literacy skill test can also be used to measure student’s skill with very low to excellentcategories

Keywords

Digital Literacy Skill, Open-ended Test, Reliability

References

• Adams, R. ., & Kho, S.-T. (1996). Acer Quest version 21. Victoria: The Australian Council forEducational Research.Ainley, J., Schulz, W., & Fraillon, J. (2016). A global measure of digital and ICT literacy skills. Global Education Monitoring Report, Education for People and Planet: Creating Sustainable Futures for All.Al-Qallaf, C. L., & Al-Mutairi, A. S. R. (2016). Digital literacy and digital content supports learning. The Electronic Library. https://doi.org/10.1108/el-05-2015-0076Alexander, B., Adams Becker, S., Cummins, M. (2016). Digital literacy. An NMC Horizon Project Strategic Brief. Scientific American. https://doi.org/10.1038/scientificamerican0995-190Alkali, Y. E., & Amichai-Hamburger, Y. (2004). Experiments in digital literacy. CyberPsychology & Behavior. https://doi.org/10.1089/cpb.2004.7.421Ayre, C., &Scally, A. J. (2014). Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development. https://doi.org/10.1177/0748175613513808Bai, X., & Ola, A. (2017). A toolfor performıng item analysıs to enhance teachıng and learnıng experıences. Issues in Information Systems, 18(1), 128–136.Bawden, D. (2008). Origins and concepts of digital literacy. In: Lankshear C and Knobel M Digital Literacies: Concepts, Policies,and Practices. New York: Peter Lang.Bekker, T., Bakker, S., Douma, I., van der Poel, J., & Scheltenaar, K. (2015). Teaching children digital literacy through design-based learning with digital toolkits in schools. International Journal of Child-Computer Interaction. https://doi.org/10.1016/j.ijcci.2015.12.001Bjørgen, A. M., & Erstad, O. (2015). The connected child: tracing digital literacy from school to leisure. Pedagogies. https://doi.org/10.1080/1554480X.2014.977290Bollard, L. M., Kerry, J. T., Whitney, S. J., & Fidock, J. J. T. (2014). Digital literacy in the Australian and New Zealand Defence Forces: Current levels and implications. In SimTecT.Brown, J. D. (2002). The Cronbach alpha reliability estimate. Shiken: JALT Testing & Evaluation SIG Newsletter.Buckingham, D. (2015). Defining digital literacy: What do young people need to know about digital media? Nordic Journal of Digital Literacy.Chan, B. S. K., Churchill, D., & Chiu, T. K. F. (2017). Digital literacy learning ın higher education through digital storytelling approach. Journal of International Education Research (JIER). https://doi.org/10.19030/jier.v13i1.9907. Cheng, P. G. F., Ramos, R. M., Bitsch, J. A., Jonas, S. M., Ix, T., See, P. L. Q., & Wehrle, K. (2016). Psychologist in a pocket: Lexicon development and content validation of a mobile-based app for depression screening. JMIR MHealth and UHealth. https://doi.org/10.2196/mhealth.5284Coffin Murray, M., & Pérez, J. (2017). Unraveling the digital literacy paradox: How higher educationfails at the fourth literacy. Issues in Informing Science and Information Technology. https://doi.org/10.28945/1982Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika. https://doi.org/10.1007/BF02310555Departmentof Elearning(2015). Green paper: Digital literacy. Retrieved from https://dge.mec.pt/sites/default/files/ERTE/Estudos_Tecnologias/elc_digital_literacy.pdfEl Masri, Y. H., Ferrara, S., Foltz, P. W., & Baird, J. A. (2017). Predicting item difficulty of science national curriculum tests: the case of key stage 2 assessments. Curriculum Journal. https://doi.org/10.1080/09585176.2016.1232201Frydenberg, M. (2015). Achieving digital literacy through game development: an authentic learning experience. Interactive Technology and Smart Education. https://doi.org/10.1108/ITSE-08-2015-0022George, D., & Mallery, P. (2003). SPSS for Windows Step by Step: A Simple Guide and Reference 17.0 Update. 2010. https://doi.org/9780335262588Greene, J. A., Yu, S. B., & Copeland,D. Z. (2014). Measuring critical components of digital literacy and their relationships with learning. Computers and Education. https://doi.org/10.1016/j.compedu.2014.03.008Güneş, E., & Bahçivan, E. (2018). A mixed research-based model for pre-service science teachers’ digital literacy: Responses to ‚which beliefs‛ and ‚how and why they interact‛ questions. Computers and Education. https://doi.org/10.1016/j.compedu.2017.11.012Gurel, D. K., Eryilmaz, A., & McDermott, L. C. (2015). A review and comparison of diagnostic instruments to identify students’ misconceptions in science. Eurasia Journal of Mathematics, Science and Technology Education. https://doi.org/10.12973/eurasia.2015.1369aHatlevik, O. E., & Christophersen, K.-A. (2013). Digital competence at the beginning of upper secondary school: Identifying factors explaining digital inclusion. Computers & Education, 63, 240–247. https://doi.org/:10.1016/j.compedu.2012.11.015Henson, R. K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34(3), 177–190.Jackson, K. M., & Trochim, W. M. K. (2002). Concept mapping as an alternative approach for the analysis of open-ended survey responses. Organizational Research Methods. https://doi.org/10.1177/109442802237114Jenkin, H., Clinton, K., Purushotma, R., Robison, A., & Weigel, M. (2006). Confronting the challenges of participatory culture: Media education for the 21th century. Retrieved from https://www.macfound.org/media/article_pdfs/JENKINS_WHITE_PAPER.PDFJosie, J., Fang, C., Chetty, K., Qigui, L., Gcora, N., & Wenwei, L. (2018). Bridging the digital divide: measuring digital literacy. Economics: The Open-Access, Open-Assessment E-Journal. https://doi.org/10.5018/economics-ejournal.ja.2018-23Karpati, A. (2011). Digital literacy in eduaction. Moscow: UNESCO Institute.Karras, D. J. (1997). Statistical methodology: II. Reliability and validity assessment in study design, Part A. Academic Emergency Medicine. https://doi.org/10.1111/j.1553-2712.1997.tb03646.xKayes, D. C. (2005). Internal validity and reliability of Kolb’s learning style inventory version 3 (1999). Journal of Business and Psychology. https://doi.org/10.1007/s10869-005-8262-4Khairani, A. Z., & Shamsuddin, H. (2016). Assessing Item Difficulty and Discrimination Indices of Teacher-Developed Multiple-Choice Tests. In Assessment for Learning Within and Beyond the Classroom. https://doi.org/10.1007/978-981-10-0908-2_35Klavir, R., & Hershkovitz, S. (2014). Teaching and evaluating ‘open -ended’ problems. ResearchGate.Kong, S. C. (2014). Developing information literacy and critical thinking skills through domain knowledge learning in digital classrooms: An experience of practicing flipped classroom strategy. Computers and Education. https://doi.org/10.1016/j.compedu.2014.05.009LAWSHE, C. H. (1975). A quantitative approach to content validity.Personnel Psychology. https://doi.org/10.1111/j.1744-6570.1975.tb01393.xLee, S.-H. (2014).Digital Literacy Education for the development of digital literacy. International Journal of Digital Literacy and Digital Competence. https://doi.org/10.4018/ijdldc.2014070103. Lee, W., & Lutz, B. D. (2016). An anchored open-ended survey approach in multiple case study analysis. In ASEE Annual Conference & Exposition. New Orleans: Louisiana. Retrieved from https://peer.asee.org/26566Literat, I. (2014). Measuring new media literacies: Towards the Development of a comprehensive assessment tool. Journal of Media Literacy Education.Loukina, A., Yoon, S., Sakano, J., Wei, Y., & Sheehan, K. M. (2016). Textual complexity as a predictor of difficulty of listening items in language proficiency tests. In Proceedings of COLING 2016, the 26th International Conferenceon Computational Linguistics: Technical Papers.Machala, D., & Orešković, M. (2014). Measuring ınformation and digital literacy activitiesthrough Learning record store repository of the national training centre for continuing education for librarians in Croatia. https://doi.org/10.1007/978-3-319-14136-7_61Matlock-Hetzel, S. (1997). Basic Concepts in item and test analysis.Annual Meeting of the Southwest Educational Research Association.Meyers, E. M., Erickson, I., & Small, R. V. (2013). Digital literacy and informal learning environments: An introduction. Learning, Media and Technology. https://doi.org/10.1080/17439884.2013.783597Mitana, J. M. ., Muwagga, A. ., & Sempala, C. (2019). Assessment for successful ıntelligence: A paradigm shift in classroom practice.International Journal of Educational Research Review, 4(1), 106–115. https://doi.org/10.24331/ijere.490162Ng, W. (2012). Can we teach digital natives digital literacy? Computers and Education. https://doi.org/10.1016/j.compedu.2012.04.016Pangrazio, L. (2016). Reconceptualising critical digital literacy. Discourse. https://doi.org/10.1080/01596306.2014.942836Perkins, K., & Frank, E. (2018). An item analysis and a reliability estimate of a classroom kinesiology achievement test.Online Submission-ERIC.Popping, R. (2015). Analyzing open-ended questions by means of text analysis procedures.BMS Bulletin of Sociological Methodology/ Bulletin de Methodologie Sociologique. https://doi.org/10.1177/0759106315597389Porat, E., Blau, I., & Barak, A. (2018). Measuring digital literacies: Junior high-school students’ perceived competencies versus actual performance. Computers and Education. https://doi.org/10.1016/j.compedu.2018.06.030Prabhu, M. T. (2010). New test measures students’ digital literacy skills. ESchool News.Prior, D. D., Mazanov, J., Meacheam, D., Heaslip, G., & Hanson, J. (2016). Attitude, digital literacy and self efficacy: Flow-on effects for online learning behavior. Internet and Higher Education. https://doi.org/10.1016/j.iheduc.2016.01.001Redmond, T. (2015). Medialiteracy ıs common sense: bridging common core standards with the media experiences of digital learners. Middle School Journal. https://doi.org/10.1080/00940771.2015.11461910Reynolds, R. (2016). Defining, designing for, and measuring ‚social constructivist digital literacy‛ development in learners: a proposed framework. Educational Technology Research and Development. https://doi.org/10.1007/s11423-015-9423-4Saputra, Maskhur DwiJoyoatmojo, S., & Wardani, D. K. (2018). The assessment of critical-thinking-skill tests for accounting students of vocational high schools.Int Ernational Journal of Educational Research Review, 3(4), 85–96. https://doi.org/10.24331/ijere.453860Schonlau, M., &Couper, M. P. (2016). Semi-automated categorization of open-ended questions. Doi.Org. https://doi.org/10.18148/srm/2016.v10i2.6213Setyawarno, D. (2016). Analysis data using program quest.Yogyakarta: Yogyakarta State University.Siddiq, F., Gochyyev, P.,& Wilson, M. (2017). Learning in digital networks–ICT literacy: A novel assessment of students’ 21st century skills. Computers and Education. https://doi.org/10.1016/j.compedu.2017.01.014Tang, C. M., & Chaw, L. Y. (2016). Digital literacy: A prerequisite for effective learning in a blended learning environment? Electronic Journal of E-Learning.Tasca, G. A., Cabrera, C., Kristjansson, E., MacNair-Semands, R., Joyce, A. S., & Ogrodniczuk, J. S. (2016). The therapeutic factor inventory-8: Using item response theory to create a brief scale for continuous process monitoring for group psychotherapy. Psychotherapy Research. https://doi.org/10.1080/10503307.2014.963729