Turkish adaptation of the digital literacy scale: A rasch analysis

Year 2025, Volume: 12 Issue: 1, 93 - 112

Hongwei Yang , Müslim Alanoğlu , Songül Karabatak , Kelly Bradley

https://doi.org/10.21449/ijate.1545220

Abstract

The study took a Rasch measurement theory approach to validating the 10-item Digital Literacy Scale (DLS) using the unidimensional rating scale model (RSM). To that end, the study used the data from a sample of online Turkish university students. The study began the Rasch analysis with all 10 items in the scale and, to improve in the local independence assumption, identified and eliminated two items which did not adequately fit the RSM. Under the eight-item DLS, the assumptions of undimensionality and local independence were both satisfied and the fit of all individual items to the RSM was adequate. Next, the psychometric properties of the eight-item DLS were examined including rating scale effectiveness, relative endorsability of the items, differential item functioning (DIF) by each of three demographic variables: (a) gender, (b) connection device, and (c) grade level. Through the analysis, evidence of reliability and validity was identified which generally supports the use of the DLS instrument among the population of online Turkish university students from which the sample was obtained. The study also identified items which demonstrated either misfit to the model or DIF by the demographic variables, and recommends they be further reviewed and revised for future use.

Keywords

Differential item functioning, Digital literacy scale, Scale validation, Rasch analysis, University students

Ethical Statement

Firat University Social Sciences and Humanities Ethics Committee, 23.05.2023-16088.

References

• Adroher, N.D., Prodinger, B., Fellinghauer, C.S., & Tennant, A. (2018) All metrics are equal, but some metrics are more equal than others: A systematic search and review on the use of the term metric'. PloS ONE, 13(3), e0193861. https://doi.org/10.1371/journal.pone.0193861
• Ala Mutka, K. (2011). Mapping digital competence: Towards a conceptual understanding (Technical Note: JRC 67075). https://doi.org/10.13140/RG.2.2.18046.00322
• Amin, H., Malik, M.A., & Akkaya, B. (2022). Development and validation of Digital Literacy Scale (DLS) and its Implication for higher education. International Journal of Distance Education and E Learning, 7(1), 24 43. https://doi.org/10.36261/ijdeel.v7i1.2224
• Andrich, D., & Marais, I. (2019). A course in Rasch measurement theory: Measuring in the educational, social and health sciences. Springer.
• Anthonysamy, L., Koo, A.C., & Hew, S.H. (2020). Self-regulated learning strategies in higher education: Fostering digital literacy for sustainable lifelong learning. Education and Information Technologies, 25, 2393-2414. https://doi.org/10.1007/s10639-020-10201-8
• Anwar, Z., Hanurawan, F., Chusniyah, T., & Setiyowati, N. (2023). Adaptation of the Academic Digital Literacy Scale for college students: A validity and reliability study. Psychological Science and Education, 28(4), 98 111. https://doi.org/10.17759/pse.2023280406
• Aryadoust, V., Ng, L.Y., & Sayama, H. (2021). A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research. Language Testing, 38(1), 6-40. https://doi.org/10.1177/0265532220927487
• Aydınlar, A., Mavi, A., Kütükçü, E., Kırımlı, E.E., Alış, D., Akın, A., & Altıntaş, L. (2024). Awareness and level of digital literacy among students receiving health-based education. BMC Medical Education, 24-38. https://doi.org/10.1186/s12909-024-05025-w
• Bartholomew, D.J., Steele, F., Moustaki, I., & Galbraith, J. (2008). Analysis of multivariate social science data (2nd ed.). Routledge.
• Bond, T.G., & Fox, C.M. (2015). Applying the Rasch model (3rd ed.). Routledge.
• Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press.
• Cameron, I.M., Scott, N.W., Adler, M., & Reid, I.C. (2014). A comparison of three methods of assessing Differential Item Functioning (DIF) in the hospital anxiety depression scale: Ordinal logistic regression, Rasch analysis and the Mantel chi-square procedure. Quality of Life Research, 23(10), 2883-2888. https://doi.org/10.1007/s11136-014-0719-3
• Cape, J., Whittington, C., Buszewicz, M., & Underwood, L. (2010). Brief psychological therapies for anxiety and depression in primary care: Meta-analysis and meta-regression. BMC Medicine, 8, 38. https://doi.org/10.1186/1741-7015-8-38
• Capinding, A.T. (2024). Development and validation of instruments for assessing the impact of artificial intelligence on students in higher education. International Journal of Educational Methodology, 10(2), 997 – 1011. https://doi.org/10.12973/ijem.10.2.997
• Ceylan, G., Eken, M.O., Yuruk, S., & Emir, F. (2023). Examining the influence of self-esteem and digital literacy on professional competence factors in dental education: A cross sectional study. Applied Sciences, 13(16), 9411. https://doi.org/10.3390/app13169411
• Chandra, S., Ghadei, K., Chennamadhava, M., & Ali, W. (2024). Development and validation of a farmer’s focused Digital Literacy Scale. Indian Journal of Extension Education, 60(1), 111-115. https://doi.org/10.48165/IJEE.2024.601RT1
• Chen, F.F., Hayes, A., Carver, C.S., Laurenceau, J.P., & Zhang, Z. (2012). Modeling general and specific variance in multifaceted constructs: A comparison of the bifactor model to other approaches. Journal of Personality, 80(1), 219–251. https://doi.org/10.1111/j.1467-6494.2011.00739.x
• Da Dalt, L., Anselmi, P., Bressan, S., Carraro, S., Baraldi, E., Robusto, E., & Perilongo, G. (2013). A short questionnaire to assess pediatric resident’s competencies: the validation process. Italian Journal of Pediatrics, 39(41), 1 8. https://doi.org/10.1186/1824-7288-39-41
• Da Dalt, L., Anselmi, P., Furlan, S., Carraro, S., Baraldi, E., Robusto, E. & Perilongo, G. (2015). Validating a set of tools designed to assess the perceived quality of training of pediatric residency programs. Italian Journal of Pediatrics, 41(2), 1 7. https://doi.org/10.1186/s13052-014-0106-2
• Durak, H.Y., & Seferoğlu, S.S. (2020). Antecedents of social media usage status: Examination of predictiveness of digital literacy, academic performance, and fear of missing out variables. Social Science Quarterly, 101(3), 1056-1074. https://doi.org/10.1111/ssqu.12790
• Erol, S., & Aydin, E. (2021). Digital literacy status of Turkish teachers. International Online Journal of Educational Sciences, 13(2), 620 633. https://doi.org/10.15345/iojes.2021.02.020
• Fan, J., & Bond T. (2019). Applying Rasch measurement in language assessment: Unidimensionality and local independence. In V. Aryadoust & M. Raquel (Eds.), Quantitative data analysis for language assessment, Vol. I: Fundamental techniques (pp. 83–102). Routledge. https://doi.org/10.4324/9781315187815
• Frampton, G.K., & Shepherd, J. (2011). Patient-reported outcomes in clinical trials of inhaled asthma medications: Systematic review and research needs. Quality of Life Research, 20, 343–357. https://doi.org/10.1007/s11136-010-9750-1
• Garzon, J., & Garzon, J. (2023). Teachers’ digital literacy and self-efficacy in blended learning. International Journal of Multidisciplinary Educational Research and Innovation. 1 (4), 162-174. https://doi.org/10.17613/cmjv-1386
• Gignac, G.E., & Kretzschmar, A. (2017). Evaluating dimensional distinctness with correlated-factor models: Limitations and suggestions. Intelligence, 62, 138 147. https://doi.org/10.1016/j.intell.2017.04.001
• Gillen, J., & Barton, D. (2010). Digital literacies: A research briefing by the technology enhanced learning phase of the Teaching and Learning Research Programme. London Knowledge Lab, Institute of Education.
• Gilster, P. (1997). Digital literacy. John Wiley & Sons.
• Gocen, A., & Sen, S. (2021). A validation of Servant Leadership Scale on multinational sample. Psychological Reports, 124(2), 752 770. https://doi.org/10.1177/0033294120957246
• Gutiérrez-Ángel, N., Sánchez-García, J.-N., Mercader-Rubio, I., García-Martín, J., & Brito-Costa, S. (2022). Digital literacy in the university setting: A literature review of empirical studies between 2010 and 2021. Frontiers in Psychology, 13, 896800. https://doi.org/10.3389/fpsyg.2022.896800
• Hagquist, C. (2019). Explaining differential item functioning focusing on the crucial role of external information–an example from the measurement of adolescent mental health. BMC Medical Research Methodology, 19(185), 1 9. https://doi.org/10.1186/s12874-019-0828-3
• Hagquist, C., Bruce, M., & Gustavsson, J.P. (2009). Using the Rasch model in nursing research: An introduction and illustrative example. International Journal of Nursing Studies, 46(3), 380-393. https://doi.org/10.1016/j.ijnurstu.2008.10.007
• Hamilton, C.B., & Chesworth, B.M. (2013). A Rasch-validated version of the upper extremity functional index for interval-level measurement of upper extremity function. Physical Therapy, 93(11), 1507–1519. https://doi.org/10.2522/ptj.20130041
• Hamutoğlu, N.B., Canan Güngören, Ö., Kaya Uyanık, G., & Gür Erdoğan, D. (2017). Adapting digital literacy scale into Turkish. Ege Journal of Education, 18(1), 408–29. https://dergipark.org.tr/tr/download/article-file/326304
• Horn, J.L., & McArdle, J.J. (1992). A practical and theoretical to measurement invariance in Aging research. Experimental Aging Research, 18, 117 144. https://doi.org/10.1080/03610739208253916
• Hwang, H.S., Zhu, L.C., & Cui, Q. (2023). Development and validation of a Digital Literacy Scale in the artificial intelligence era for college students. KSII Transactions on Internet and Information Systems, 17(8), 2241-2258. https://doi.org/10.3837/tiis.2023.08.016
• Lazonder, A.W., Walraven, A., Gijlers, H., & Janssen, N. (2020). Longitudinal assessment of digital literacy in children: Findings from a large Dutch single-school study. Computers & Education, 143, 103681. https://doi.org/10.1016/j.compedu.2019.103681
• Lee, Y.W. (2004). Examining passage-related local item dependence (LID) and measurement construct using Q3 statistics in an EFL reading comprehension test. Language Testing, 21(1), 74–100. https://doi.org/10.1191/0265532204lt260oa
• Linacre, J.M. (n.d.). Table 23.0 Variance components for items. https://www.winsteps.com/winman/table23_0.htm
• Linacre, J.M. (2018, September 2). Detecting multidimensionality in Rasch data using Winsteps Table 23 [Video]. Youtube. https://www.youtube.com/watch?v=sna19QemE50
• Linacre, J.M. (2023). Winsteps® Rasch measurement computer program user's guide. Version 5.6.4. Winsteps.com
• Liu, Y., Millsap, R.E., West, S.G., Tein, J.-Y., Tanaka, R., & Grimm, K.J. (2017). Testing measurement invariance in longitudinal data with ordered-categorical measures. Psychological Methods, 22(3), 486–506. https://doi.org/10.1037/met0000075
• Liza, K., & Andriyanti, E. (2020). Digital literacy scale of English pre-service teachers and their perceived readiness toward the application of digital technologies. Journal of Education and Learning, 14(1), 74-79. https://doi.org/10.11591/edulearn.v14i1.13925
• Meredith, W. (1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58, 525-543. https://doi.org/10.1007/BF02294825
• Miot, H.A. (2020). Analysis of ordinal data in clinical and experimental studies. Jornal Vascular Brasileiro, 19, e20200185. https://doi.org/10.1590/1677-5449.200185
• Mundfrom, D.J., Shaw, D.G., & Ke, T.L. (2005). Minimum sample size recommendations for conducting factor analyses. International Journal of Testing, 5(2), 159–168. https://doi.org/10.1207/s15327574ijt0502_4
• Ng, W. (2012a). Can we teach digital natives digital literacy?. Computers & Education, 59(3), 6881065-1078. https://doi.org/10.1016/j.compedu.2012.04.016
• Ng, W. (2012b). Empowering scientific literacy through digital literacy and multiliteracies. Nova Science Publishers, Inc.
• Nguyen, L.A.T., & Habók, A. (2024). Tools for assessing teacher digital literacy: A review. Journals of Computers in Education, 11, 305-346). https://doi.org/10.1007/s40692-022-00257-5
• Ningsih, S.K., Mulyono, H., Rahmah, R.A., & Fitriani, N.A. (2021). A Rasch-based validation of EFL teachers’ received online social support scale. Cogent Education, 8(1), 1-13. https://doi.org/10.1080/2331186X.2021.1957529
• Noorrizki, R.D., Abadi, D., Siwi, N.S.W., Sa'id, M., Mantara, A.Y., & Ramadhani, F. (2022). Factors affecting digital literacy in young adults. Proceedings of the International Conference of Psychology 2022 (ICoPsy 2022), 308-315, KnE Social Sciences. https://doi.org/10.18502/kss.v7i18.12396
• Nunnally, J.C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
• Olur, B., & Ocak, G. (2021). Digital literacy self-efficacy scale: A scale development study. African Educational Research Journal, 9(2), 581 590. https://doi.org/10.30918/AERJ.92.21.074
• Pangrazio, L., Godhe, A.-L., & Ledesma, A.G.L. (2020). What is digital literacy? A comparative review of publications across three language contexts. E-Learning and Digital Media, 17(6), 442-459. https://doi.org/10.1177/2042753020946291
• Perera, M.U., Gardner, L., & Peiris, A. (2016). Investigating the Interrelationship between undergraduates’ digital literacy and self-regulated learning skills. Proceedings of the 2016 International Conference on Information Systems (ICIS), Dublin, Ireland.
• Reddy, P., Chaudhary, K., Sharma, B., & Chand, D. (2021). Contextualized game-based intervention for digital literacy for the Pacific Islands. Education and Information Technologies, 26, 5535–5562. https://doi.org/10.1007/s10639-021-10534-y
• Reddy, P., Chaudhary, K., Sharma, B., & Hussein, S. (2023). Essaying the design, development and validation processes of a new digital literacy scale. Online Information Review, 47(2), 371-397. https://doi.org/10.1108/oir-10-2021-0532
• Reddy, P., Chaudhary, K., & Hussein, S. (2023). A digital literacy model to narrow the digital literacy skills gap. Heliyon, 9, e14878. https://doi.org/10.1016/j.heliyon.2023.e14878
• Reise, S.P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667–696. https://doi.org/10.1080/00273171.2012.715555
• Reise, S.P., Morizot, J., & Hays, R.D. (2007). The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Quality of Life Research, 16(1), 19–31. https://doi.org/10.1007/s11136-007-9183-7
• Rodriguez, A., Reise, S.P., & Haviland, M.G. (2016a). Applying bifactor statistical indices in the evaluation of psychological measures. Journal of Personality Assessment, 98(3), 223–237. https://doi.org/10.1080/00223891.2015.1089249
• Rodriguez, A., Reise, S.P., & Haviland, M.G. (2016b). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137–150. https://doi.org/10.1037/met0000045
• Setiyowati N., & Razak, A.Z.A. (2020). Perceived leadership styles and academicians’ job performance: Teaching, research, and community services in Indonesia. Malaysia Online Journal of Psychology and Counseling, 7(1), 11 26. https://jupidi.um.edu.my/index.php/MOJC/article/view/24179/11654
• Skrondal, A., & Rabe-Hesketh, S. (2004). Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models. Chapman Hall/CRC.
• Tor, D.L., Başaran, S.D., & Arık, E. (2022). Examining the digital Literacy level of teacher candidates. Journal of Kırşehir Education Faculty, 23(2), 2027 2064. https://doi.org/10.29299/kefad.1047590
• UNESCO. (2018). A global framework of reference on digital literacy skills for indicator 4.4.2. https://uis.unesco.org/sites/default/files/documents/ip51-global-framework-reference-digital-literacy-skills-2018-en.pdf
• Urbancikova, N., Manakova, N., & Ganna, B. (2017). Socio-economic and regional factors of digital literacy related to prosperity. Quality Innovation Prosperity, 21(2), 124-141. https://doi.org/10.12776/qip.v21i2.942
• Üstündağ, M.T., Güneş, E., & Bahçivan, E. (2017). Turkish adaptation of Digital Literacy Scale and investigating pre-service science teachers' digital literacy. Journal of Education and Future, 12, 19-29. https://dergipark.org.tr/tr/download/article-file/332115
• Wright, B.D. (1996). Local dependency, correlations and principal components. Rasch Measurement Transactions, 10(3), 509–511. https://www.rasch.org/rmt/rmt103b.htm
• Wu, M., DeWitt1, D., Alias, N., & Nazry, N.N.M. (2022). Digital literacy from 2018-2021: A scientometric study of the literature. https://doi.org/10.21203/rs.3.rs-1848300/v1
• Yen, W.M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145. https://doi.org/10.1177/014662168400800201
• Zwick, R., Thayer, D.T., & Lewis, C. (1999). An empirical Bayes approach to Mantel-Haenszel DIF analysis. Journal of Educational Measurement, 36(1), 1 28. https://doi.org/10.1111/j.1745-3984.1999.tb00543.x
• Zwick, R. (2012). A review of ETS Differential Item Functioning assessment procedures: Flagging rules, minimum sample size requirements, and criterion refinement. ETS Research Report Series, i-30. https://doi.org/10.1002/j.2333-8504.2012.tb02290.x