Development and validation of digital assessment literacy self-efficacy scale

Year 2025, Volume: 12 Issue: 3, 806 - 824, 04.09.2025

Ruhan Karadağ Yılmaz , İlhan Koyuncu

https://doi.org/10.21449/ijate.1606780

Abstract

This study aimed to develop a scale to measure teachers' digital assessment literacy self-efficacy. Teachers were selected from all regions of Turkey and different branches at primary, secondary, and high school levels to enhance generalization and diversity. The data was collected from 314 teachers for the exploratory factor analysis and 296 for the confirmatory factor analysis. Various evidence was obtained regarding the validity and reliability of the Digital Assessment Literacy Self-Efficacy Scale (DALSS). The exploratory factor analysis results revealed that DALSS was found to have a 3-factor structure with 24 items. The dimensions were named as “Using appropriate tool and assessment type”, “Feedback and developing appropriate tools for the objectives and levels,” and “Preparing exams and evaluating the usability of the scores.” Confirmatory factor analysis (CFA) results showed that the hypothetical structure of the scale fit the data. Second-order CFA also confirmed the 3-factor structure of digital assessment literacy self-efficacy. Convergent and divergent validity results proved evidence for construct validity. Analyses based on known-group validity revealed that the DALSS is a discriminative instrument. The internal consistency reliability values of DALSS and its factors were found to vary between .96 and .98. The findings for practice revealed that although DALSS scores did not differentiate according to gender, professional experience and teaching level, they significantly differed in terms of teaching fields.

Keywords

Assessment literacy , Digital literacy , Digital assessment , Scale development , Self-efficacy

Ethical Statement

Necmettin Erbakan University Social Sciences and Humanities Scientific Research Ethics Committee, 12603 (Approval date: 20/01/2023).

References

• American Federation of Teachers [AFT], National Council on Measurement in Education [NCME], & National Education Association [NEA] (1990). Standards for teacher competence in educational assessment of students. Educational Measurement: Issues and Practice, 9(4), 30-32.
• Al-Bahlani, S.M. (2019). Assessment literacy: A study of EFL teachers' assessment knowledge, perspectives, and classroom behaviors [Doctoral dissertation, The University of Arizona].  Available from ProQuest Dissertations & Theses Global (2229389562). https://www.proquest.com/dissertations-theses/assessment-literacy-study-efl-teachers-knowledge/docview/2229389562/se-2
• Al-Bahlani, S.M., & Ecke, P. (2023). Assessment competence and practices including digital assessment literacy of postsecondary English language teachers in Oman. Cogent Education, 10(2), 2239535. https://doi.org/10.1080/2331186X.2023.2239535
• Anderson, J.C., & Gerbing, D.W. (1984). The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis. Psychometrika, 49(2), 155-173. https://doi.org/10.1007/BF02294170
• Boateng, G.O., Neilands, T.B., Frongillo. E.A., Melgar-Quiñonez. H.R., & Young, S.L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Front Public Health. 11(6), 1 18. https://doi.org/10.3389/fpubh.2018.00149
• Browne, M.W., & Cudeck, R. (1993). Alternative ways of assessing model fit. Sage Focus Editions, 154, 136-136. https://doi.org/10.1177/0049124192021002005
• Cabezas-Gonzalez, M., Casillas-Martín, S., Sanches-Ferreira, M., & Teixeira Diogo, F.L. (2017). Do gender and age affect the level of digital competence? A study with university students. Fonseca Journal of Communication, 15, 109 125. https://doi.org/10.14201/fjc201715109125
• Campbell, D.T., & Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81-105.
• Cattell, R.B. (1978). The scientific use of factor analysis in behavioral and life sciences. Plenum Press. https://doi.org/10.1007/978-1-4684-2262-7
• Chan, C.K.Y., & Luo, J. (2020). A four-dimensional conceptual framework for student assessment literacy in holistic competency development. Assessment & Evaluation in Higher Education, 46(3), 451-466. https://doi.org/10.1080/02602938.2020.1777388
• Comrey, A.L., & Lee, H.B. (1992). A first course in factor analysis. Lawrence Erlbaum.
• Cronbach, L.J., & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin 52(4), 281. https://doi.org/10.1037/h0040957
• DeLuca, C., & Klinger, D.A. (2010). Assessment literacy development: Identifying gaps in teacher candidates’ learning. Assessment in Education: Principles, Policy & Practice, 17(4), 419-438. https://doi.org/10.1080/0969594X.2010.516643
• DeLuca, C., LaPointe-McEwan, D., & Luhanga, U. (2016a). Approaches to classroom assessment inventory: A new instrument to support teacher assessment literacy. Educational Assessment, 21(4), 248-266. https://doi.org/10.1080/10627197.2016.1236677
• DeLuca, C., LaPointe-McEwan, D., & Luhanga, U. (2016b). Teacher assessment literacy: A review of international standards and measures. Educational Assessment, Evaluation, and Accountability, 28, 251–272. https://doi.org/10.1007/s11092-015-9233-6
• Edwards, F. (2016). A rubric to track the development of secondary pre-service and novice teachers’ summative assessment literacy. Assessment in Education: Principles, Policy & Practice, 24(2), 1-23. https://doi.org/10.1080/0969594X.2016.1245651
• Erkuş, A. (2012). Psikolojide ölçme ve ölçek geliştirme [Measurement and scale development in Psychology]. Pegem Akademi.
• Eyal, L. (2012). Digital assessment literacy - the core role of the teacher in a digital environment. Educational Technology & Society, 15(2), 37 49. https://www.jstor.org/stable/jeductechsoci.15.2.37
• Fraenkel, J.R., Wallen, N.E., & Hyun, H.H. (2011). How to design and evaluate research in education. McGraw-Hill.
• Fornell, C., & Larcker, D.F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18, 39–50.
• Fulcher, G. (2012). Assessment literacy for the language classroom. Language Assessment Quarterly, 9, 113–132. https://doi.org/10.1080/15434303.2011.642041
• Hair, J.F., Black, W.C., Babin, B.J., & Anderson, R.E. (2009). Multivariate data analysis. Prentice-Hall.
• Hu, L.T., & Bentler, P.M. (1999). Cut-off criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. https://doi.org/10.1080/10705519909540118
• Kline, R.B. (2016). Principle and practice of structural equation modeling. The Guilford Press.
• Koyuncu, I., & Kılıç, A. (2019). The use of exploratory and confirmatory factor analyses: A document analysis. Education and Science, 44(198), 361-388.
• Lawshe, C.H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563-575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x
• Lucas, M., Bem-Haja, P., Siddiq, F., Moreira, A., & Redecker, C. (2021). The relation between in-service teachers’ digital competence and personal and contextual factors: What matters most? Computers & Education, 160, 104052. https://doi.org/10.1016/j.compedu.2020.104052.
• Mertler, C.A. (2004). Secondary teachers' assessment literacy: Does classroom experience make a difference? American Secondary Education, 33(1), 49 64. https://www.jstor.org/stable/41064623
• Mertler, C.A., & Campbell, C.S. (2005). Measuring teachers’ knowledge and application of classroom assessment concepts: Development of the assessment literacy inventory (Paper presentation). The annual meeting of the American Educational Research Association, Montreal, Quebec, Canada.
• Ministry of National Education [MoNE] (2017). General qualifications for teaching profession. General Directorate of Teacher Training and Development. https://oygm.meb.gov.tr/dosyalar/StPrg/Ogretmenlik_Meslegi_Genel_Yeterlikleri.pdf
• Mindrila, D. (2010). Maximum likelihood (ML) and diagonally weighted least squares (DWLS) estimation procedures: A comparison of estimation bias with ordinal and multivariate non-normal data. International Journal of Digital Society, 1(1), 60-66.
• Morgado, F.F., Meireles, J.F., Neves, C.M., Amaral, A.C., & Ferreira, M.E. (2018). Scale development: ten main limitations and recommendations to improve future research practices. Psicologia: Reflexão e Crítica, 30(3), 1-20. https://doi.org/10.1186/s41155-016-0057-1
• Nimehchisalem, V., & Bhatti, N. (2019). A review of literature on language assessment literacy in last two decades (1999-2018). International Journal of Innovation, Creativity and Change, 8(11), 44-59.
• Nyagi, K., & Rajendran, M. (2020). Pre-service teachers’ approaches to classroom assessment. Humanities & Social Sciences Reviews, 8(1), 666 673. https://doi.org/10.18510/hssr.2020.8180
• Plake, B.S., & Impara, J.C. (1992). Teacher competencies questionnaire description. University of Nebraska.
• Popham, W.J. (2011). Assessment literacy overlooked: A teacher educator’s confession. The Teacher Educator, 46, 265–273. https://doi.org/10.1080/08878730.2011.605048
• R Core Team (2023). R: A Language and environment for statistical computing. (Version 4.3) [Computer software]. https://cran.r-project.org. (R packages retrieved from CRAN snapshot 2024-01-09).
• Rezai, A., Alibakhshi, G., Farokhipour, S., & Miri, M. (2021). A phenomenographic study on language assessment literacy: hearing from Iranian university teachers. Language Testing in Asia, 11(26), 1-25. https://doi.org/10.1186/s40468-021-00142-5
• Rogier, D. (2014). Assessment literacy: Building a base for better teaching and learning. English Teaching Forum, 52(3), 2 13. https://americanenglish.state.gov/files/ae/resource_files/etf_52_3_02-13.pdf
• Rosseel, Y. (2019). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. https://doi.org/10.18637/jss.v048.i02
• Shams, J.A., Iqbal, M.Z., & Iqbal, M.Z. (2018). Investigation of classroom assessment literacy of university teachers of Punjab. Pakistan Journal of Distance and Online Learning, IV(II), 103-118. https://files.eric.ed.gov/fulltext/EJ1267033.pdf
• Stiggins, R. (1991). Assessment literacy. The Phi Delta Kappan, 72, 534–539. https://learnline.cdu.edu.au/commonunits/documents/Scaffolding%20and%20formative%20assessment.pdf
• Stiggins, R.J. (1999). Evaluating classroom assessment training in teacher education programs. Educational Measurement: Issues and Practice, 18(1), 23 17. https://doi.org/10.1111/j.1745-3992.1999.tb00004.x
• Tabachnick, B.G., & Fidell, L.S. (2012). Using multivariate statistics. Allyn & Bacon.
• The jamovi project (2024). Jamovi (Version 2.5) [Computer Software]. https://www.jamovi.org
• Ullman, J.B. (2012). Structural equation modeling. In B.G. Tabachnik & L.S. Fidell (Eds.), Using multivariate statistics (6th ed.). Pearson.
• Volante, L., & Fazio, X. (2007). Exploring teacher candidates' assessment literacy: Implications for teacher education reform and professional development. Canadian Journal of Education, 30(3), 749-770. https://eric.ed.gov/?id=EJ780818
• Walters, F.S. (2010). Cultivating assessment literacy: Standards evaluation through language-test specification reverse engineering. Language Assessment Quarterly, 7(4), 317-342. https://doi.org/10.1080/15434303.2010.516042
• Worthington, R.L., & Whittaker, T.A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806-838. https://doi.org/10.1177/0011000006288127
• Yamtim, V., & Wong-Wanich, S. (2014). A study of classroom assessment literacy of primary school teachers. Procedia Social and Behavioral Sciences, 116, 2998 3004. https://doi.org/10.1016/j.sbspro.2014.01.696
• Yong, A.G., & Pearce, S. (2013). A beginner’s guide to factor analysis: Focusing on exploratory factor analysis. Tutorials in Quantitative Methods for Psychology, 9(2), 79 94. https://doi.org/10.20982/tqmp.09.2.p079
• Yuan, K.H. (2009). Normal distribution based pseudo ML for missing data: With applications to mean and covariance structure analysis. Journal of Multivariate Analysis, 100(9), 1900-1918.
• Zhang, Z., & Burry-Stock, J.A. (2003). Classroom assessment practices and teachers’ self-perceived assessment skills. Applied Measurement in Education, 16(4), 323-342. https://doi.org/10.1207/S15324818AME1604_4