Turkish adaptation of the digital literacy scale: A rasch analysis

Abstract

The study took a Rasch measurement theory approach to validating the 10-item Digital Literacy Scale (DLS) using the unidimensional rating scale model (RSM). To that end, the study used the data from a sample of online Turkish university students. The study began the Rasch analysis with all 10 items in the scale and, to improve in the local independence assumption, identified and eliminated two items which did not adequately fit the RSM. Under the eight-item DLS, the assumptions of undimensionality and local independence were both satisfied and the fit of all individual items to the RSM was adequate. Next, the psychometric properties of the eight-item DLS were examined including rating scale effectiveness, relative endorsability of the items, differential item functioning (DIF) by each of three demographic variables: (a) gender, (b) connection device, and (c) grade level. Through the analysis, evidence of reliability and validity was identified which generally supports the use of the DLS instrument among the population of online Turkish university students from which the sample was obtained. The study also identified items which demonstrated either misfit to the model or DIF by the demographic variables, and recommends they be further reviewed and revised for future use.

Keywords

• Differential item functioning
• Digital literacy scale
• Scale validation
• Rasch analysis
• University students

References

1. Adroher, N.D., Prodinger, B., Fellinghauer, C.S., & Tennant, A. (2018) All metrics are equal, but some metrics are more equal than others: A systematic search and review on the use of the term metric'. PloS ONE, 13(3), e0193861. https://doi.org/10.1371/journal.pone.0193861
2. Ala Mutka, K. (2011). Mapping digital competence: Towards a conceptual understanding (Technical Note: JRC 67075). https://doi.org/10.13140/RG.2.2.18046.00322
3. Amin, H., Malik, M.A., & Akkaya, B. (2022). Development and validation of Digital Literacy Scale (DLS) and its Implication for higher education. International Journal of Distance Education and E Learning, 7(1), 24 43. https://doi.org/10.36261/ijdeel.v7i1.2224
4. Andrich, D., & Marais, I. (2019). A course in Rasch measurement theory: Measuring in the educational, social and health sciences. Springer.
5. Anthonysamy, L., Koo, A.C., & Hew, S.H. (2020). Self-regulated learning strategies in higher education: Fostering digital literacy for sustainable lifelong learning. Education and Information Technologies, 25, 2393-2414. https://doi.org/10.1007/s10639-020-10201-8
6. Anwar, Z., Hanurawan, F., Chusniyah, T., & Setiyowati, N. (2023). Adaptation of the Academic Digital Literacy Scale for college students: A validity and reliability study. Psychological Science and Education, 28(4), 98 111. https://doi.org/10.17759/pse.2023280406
7. Aryadoust, V., Ng, L.Y., & Sayama, H. (2021). A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research. Language Testing, 38(1), 6-40. https://doi.org/10.1177/0265532220927487
8. Aydınlar, A., Mavi, A., Kütükçü, E., Kırımlı, E.E., Alış, D., Akın, A., & Altıntaş, L. (2024). Awareness and level of digital literacy among students receiving health-based education. BMC Medical Education, 24-38. https://doi.org/10.1186/s12909-024-05025-w

9. Bartholomew, D.J., Steele, F., Moustaki, I., & Galbraith, J. (2008). Analysis of multivariate social science data (2nd ed.). Routledge.
10. Bond, T.G., & Fox, C.M. (2015). Applying the Rasch model (3rd ed.). Routledge.
11. Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press.
12. Cameron, I.M., Scott, N.W., Adler, M., & Reid, I.C. (2014). A comparison of three methods of assessing Differential Item Functioning (DIF) in the hospital anxiety depression scale: Ordinal logistic regression, Rasch analysis and the Mantel chi-square procedure. Quality of Life Research, 23(10), 2883-2888. https://doi.org/10.1007/s11136-014-0719-3
13. Cape, J., Whittington, C., Buszewicz, M., & Underwood, L. (2010). Brief psychological therapies for anxiety and depression in primary care: Meta-analysis and meta-regression. BMC Medicine, 8, 38. https://doi.org/10.1186/1741-7015-8-38
14. Capinding, A.T. (2024). Development and validation of instruments for assessing the impact of artificial intelligence on students in higher education. International Journal of Educational Methodology, 10(2), 997 – 1011. https://doi.org/10.12973/ijem.10.2.997
15. Ceylan, G., Eken, M.O., Yuruk, S., & Emir, F. (2023). Examining the influence of self-esteem and digital literacy on professional competence factors in dental education: A cross sectional study. Applied Sciences, 13(16), 9411. https://doi.org/10.3390/app13169411
16. Chandra, S., Ghadei, K., Chennamadhava, M., & Ali, W. (2024). Development and validation of a farmer’s focused Digital Literacy Scale. Indian Journal of Extension Education, 60(1), 111-115. https://doi.org/10.48165/IJEE.2024.601RT1
17. Chen, F.F., Hayes, A., Carver, C.S., Laurenceau, J.P., & Zhang, Z. (2012). Modeling general and specific variance in multifaceted constructs: A comparison of the bifactor model to other approaches. Journal of Personality, 80(1), 219–251. https://doi.org/10.1111/j.1467-6494.2011.00739.x
18. Da Dalt, L., Anselmi, P., Bressan, S., Carraro, S., Baraldi, E., Robusto, E., & Perilongo, G. (2013). A short questionnaire to assess pediatric resident’s competencies: the validation process. Italian Journal of Pediatrics, 39(41), 1 8. https://doi.org/10.1186/1824-7288-39-41
19. Da Dalt, L., Anselmi, P., Furlan, S., Carraro, S., Baraldi, E., Robusto, E. & Perilongo, G. (2015). Validating a set of tools designed to assess the perceived quality of training of pediatric residency programs. Italian Journal of Pediatrics, 41(2), 1 7. https://doi.org/10.1186/s13052-014-0106-2
20. Durak, H.Y., & Seferoğlu, S.S. (2020). Antecedents of social media usage status: Examination of predictiveness of digital literacy, academic performance, and fear of missing out variables. Social Science Quarterly, 101(3), 1056-1074. https://doi.org/10.1111/ssqu.12790
21. Erol, S., & Aydin, E. (2021). Digital literacy status of Turkish teachers. International Online Journal of Educational Sciences, 13(2), 620 633. https://doi.org/10.15345/iojes.2021.02.020
22. Fan, J., & Bond T. (2019). Applying Rasch measurement in language assessment: Unidimensionality and local independence. In V. Aryadoust & M. Raquel (Eds.), Quantitative data analysis for language assessment, Vol. I: Fundamental techniques (pp. 83–102). Routledge. https://doi.org/10.4324/9781315187815
23. Frampton, G.K., & Shepherd, J. (2011). Patient-reported outcomes in clinical trials of inhaled asthma medications: Systematic review and research needs. Quality of Life Research, 20, 343–357. https://doi.org/10.1007/s11136-010-9750-1
24. Garzon, J., & Garzon, J. (2023). Teachers’ digital literacy and self-efficacy in blended learning. International Journal of Multidisciplinary Educational Research and Innovation. 1 (4), 162-174. https://doi.org/10.17613/cmjv-1386
25. Gignac, G.E., & Kretzschmar, A. (2017). Evaluating dimensional distinctness with correlated-factor models: Limitations and suggestions. Intelligence, 62, 138 147. https://doi.org/10.1016/j.intell.2017.04.001
26. Gillen, J., & Barton, D. (2010). Digital literacies: A research briefing by the technology enhanced learning phase of the Teaching and Learning Research Programme. London Knowledge Lab, Institute of Education.
27. Gilster, P. (1997). Digital literacy. John Wiley & Sons.
28. Gocen, A., & Sen, S. (2021). A validation of Servant Leadership Scale on multinational sample. Psychological Reports, 124(2), 752 770. https://doi.org/10.1177/0033294120957246
29. Gutiérrez-Ángel, N., Sánchez-García, J.-N., Mercader-Rubio, I., García-Martín, J., & Brito-Costa, S. (2022). Digital literacy in the university setting: A literature review of empirical studies between 2010 and 2021. Frontiers in Psychology, 13, 896800. https://doi.org/10.3389/fpsyg.2022.896800
30. Hagquist, C. (2019). Explaining differential item functioning focusing on the crucial role of external information–an example from the measurement of adolescent mental health. BMC Medical Research Methodology, 19(185), 1 9. https://doi.org/10.1186/s12874-019-0828-3
31. Hagquist, C., Bruce, M., & Gustavsson, J.P. (2009). Using the Rasch model in nursing research: An introduction and illustrative example. International Journal of Nursing Studies, 46(3), 380-393. https://doi.org/10.1016/j.ijnurstu.2008.10.007
32. Hamilton, C.B., & Chesworth, B.M. (2013). A Rasch-validated version of the upper extremity functional index for interval-level measurement of upper extremity function. Physical Therapy, 93(11), 1507–1519. https://doi.org/10.2522/ptj.20130041
33. Hamutoğlu, N.B., Canan Güngören, Ö., Kaya Uyanık, G., & Gür Erdoğan, D. (2017). Adapting digital literacy scale into Turkish. Ege Journal of Education, 18(1), 408–29. https://dergipark.org.tr/tr/download/article-file/326304
34. Horn, J.L., & McArdle, J.J. (1992). A practical and theoretical to measurement invariance in Aging research. Experimental Aging Research, 18, 117 144. https://doi.org/10.1080/03610739208253916
35. Hwang, H.S., Zhu, L.C., & Cui, Q. (2023). Development and validation of a Digital Literacy Scale in the artificial intelligence era for college students. KSII Transactions on Internet and Information Systems, 17(8), 2241-2258. https://doi.org/10.3837/tiis.2023.08.016
36. Lazonder, A.W., Walraven, A., Gijlers, H., & Janssen, N. (2020). Longitudinal assessment of digital literacy in children: Findings from a large Dutch single-school study. Computers & Education, 143, 103681. https://doi.org/10.1016/j.compedu.2019.103681
37. Lee, Y.W. (2004). Examining passage-related local item dependence (LID) and measurement construct using Q3 statistics in an EFL reading comprehension test. Language Testing, 21(1), 74–100. https://doi.org/10.1191/0265532204lt260oa
38. Linacre, J.M. (n.d.). Table 23.0 Variance components for items. https://www.winsteps.com/winman/table23_0.htm
39. Linacre, J.M. (2018, September 2). Detecting multidimensionality in Rasch data using Winsteps Table 23 [Video]. Youtube. https://www.youtube.com/watch?v=sna19QemE50
40. Linacre, J.M. (2023). Winsteps® Rasch measurement computer program user's guide. Version 5.6.4. Winsteps.com
41. Liu, Y., Millsap, R.E., West, S.G., Tein, J.-Y., Tanaka, R., & Grimm, K.J. (2017). Testing measurement invariance in longitudinal data with ordered-categorical measures. Psychological Methods, 22(3), 486–506. https://doi.org/10.1037/met0000075
42. Liza, K., & Andriyanti, E. (2020). Digital literacy scale of English pre-service teachers and their perceived readiness toward the application of digital technologies. Journal of Education and Learning, 14(1), 74-79. https://doi.org/10.11591/edulearn.v14i1.13925
43. Meredith, W. (1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58, 525-543. https://doi.org/10.1007/BF02294825
44. Miot, H.A. (2020). Analysis of ordinal data in clinical and experimental studies. Jornal Vascular Brasileiro, 19, e20200185. https://doi.org/10.1590/1677-5449.200185
45. Mundfrom, D.J., Shaw, D.G., & Ke, T.L. (2005). Minimum sample size recommendations for conducting factor analyses. International Journal of Testing, 5(2), 159–168. https://doi.org/10.1207/s15327574ijt0502_4
46. Ng, W. (2012a). Can we teach digital natives digital literacy?. Computers & Education, 59(3), 6881065-1078. https://doi.org/10.1016/j.compedu.2012.04.016
47. Ng, W. (2012b). Empowering scientific literacy through digital literacy and multiliteracies. Nova Science Publishers, Inc.
48. Nguyen, L.A.T., & Habók, A. (2024). Tools for assessing teacher digital literacy: A review. Journals of Computers in Education, 11, 305-346). https://doi.org/10.1007/s40692-022-00257-5
49. Ningsih, S.K., Mulyono, H., Rahmah, R.A., & Fitriani, N.A. (2021). A Rasch-based validation of EFL teachers’ received online social support scale. Cogent Education, 8(1), 1-13. https://doi.org/10.1080/2331186X.2021.1957529
50. Noorrizki, R.D., Abadi, D., Siwi, N.S.W., Sa'id, M., Mantara, A.Y., & Ramadhani, F. (2022). Factors affecting digital literacy in young adults. Proceedings of the International Conference of Psychology 2022 (ICoPsy 2022), 308-315, KnE Social Sciences. https://doi.org/10.18502/kss.v7i18.12396
51. Nunnally, J.C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
52. Olur, B., & Ocak, G. (2021). Digital literacy self-efficacy scale: A scale development study. African Educational Research Journal, 9(2), 581 590. https://doi.org/10.30918/AERJ.92.21.074
53. Pangrazio, L., Godhe, A.-L., & Ledesma, A.G.L. (2020). What is digital literacy? A comparative review of publications across three language contexts. E-Learning and Digital Media, 17(6), 442-459. https://doi.org/10.1177/2042753020946291
54. Perera, M.U., Gardner, L., & Peiris, A. (2016). Investigating the Interrelationship between undergraduates’ digital literacy and self-regulated learning skills. Proceedings of the 2016 International Conference on Information Systems (ICIS), Dublin, Ireland.
55. Reddy, P., Chaudhary, K., Sharma, B., & Chand, D. (2021). Contextualized game-based intervention for digital literacy for the Pacific Islands. Education and Information Technologies, 26, 5535–5562. https://doi.org/10.1007/s10639-021-10534-y
56. Reddy, P., Chaudhary, K., Sharma, B., & Hussein, S. (2023). Essaying the design, development and validation processes of a new digital literacy scale. Online Information Review, 47(2), 371-397. https://doi.org/10.1108/oir-10-2021-0532
57. Reddy, P., Chaudhary, K., & Hussein, S. (2023). A digital literacy model to narrow the digital literacy skills gap. Heliyon, 9, e14878. https://doi.org/10.1016/j.heliyon.2023.e14878
58. Reise, S.P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667–696. https://doi.org/10.1080/00273171.2012.715555
59. Reise, S.P., Morizot, J., & Hays, R.D. (2007). The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Quality of Life Research, 16(1), 19–31. https://doi.org/10.1007/s11136-007-9183-7
60. Rodriguez, A., Reise, S.P., & Haviland, M.G. (2016a). Applying bifactor statistical indices in the evaluation of psychological measures. Journal of Personality Assessment, 98(3), 223–237. https://doi.org/10.1080/00223891.2015.1089249
61. Rodriguez, A., Reise, S.P., & Haviland, M.G. (2016b). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137–150. https://doi.org/10.1037/met0000045
62. Setiyowati N., & Razak, A.Z.A. (2020). Perceived leadership styles and academicians’ job performance: Teaching, research, and community services in Indonesia. Malaysia Online Journal of Psychology and Counseling, 7(1), 11 26. https://jupidi.um.edu.my/index.php/MOJC/article/view/24179/11654
63. Skrondal, A., & Rabe-Hesketh, S. (2004). Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models. Chapman Hall/CRC.
64. Tor, D.L., Başaran, S.D., & Arık, E. (2022). Examining the digital Literacy level of teacher candidates. Journal of Kırşehir Education Faculty, 23(2), 2027 2064. https://doi.org/10.29299/kefad.1047590
65. UNESCO. (2018). A global framework of reference on digital literacy skills for indicator 4.4.2. https://uis.unesco.org/sites/default/files/documents/ip51-global-framework-reference-digital-literacy-skills-2018-en.pdf
66. Urbancikova, N., Manakova, N., & Ganna, B. (2017). Socio-economic and regional factors of digital literacy related to prosperity. Quality Innovation Prosperity, 21(2), 124-141. https://doi.org/10.12776/qip.v21i2.942
67. Üstündağ, M.T., Güneş, E., & Bahçivan, E. (2017). Turkish adaptation of Digital Literacy Scale and investigating pre-service science teachers' digital literacy. Journal of Education and Future, 12, 19-29. https://dergipark.org.tr/tr/download/article-file/332115
68. Wright, B.D. (1996). Local dependency, correlations and principal components. Rasch Measurement Transactions, 10(3), 509–511. https://www.rasch.org/rmt/rmt103b.htm
69. Wu, M., DeWitt1, D., Alias, N., & Nazry, N.N.M. (2022). Digital literacy from 2018-2021: A scientometric study of the literature. https://doi.org/10.21203/rs.3.rs-1848300/v1
70. Yen, W.M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145. https://doi.org/10.1177/014662168400800201
71. Zwick, R., Thayer, D.T., & Lewis, C. (1999). An empirical Bayes approach to Mantel-Haenszel DIF analysis. Journal of Educational Measurement, 36(1), 1 28. https://doi.org/10.1111/j.1745-3984.1999.tb00543.x
72. Zwick, R. (2012). A review of ETS Differential Item Functioning assessment procedures: Flagging rules, minimum sample size requirements, and criterion refinement. ETS Research Report Series, i-30. https://doi.org/10.1002/j.2333-8504.2012.tb02290.x