Development and validation of digital assessment literacy self-efficacy scale

Abstract

This study aimed to develop a scale to measure teachers' digital assessment literacy self-efficacy. Teachers were selected from all regions of Turkey and different branches at primary, secondary, and high school levels to enhance generalization and diversity. The data was collected from 314 teachers for the exploratory factor analysis and 296 for the confirmatory factor analysis. Various evidence was obtained regarding the validity and reliability of the Digital Assessment Literacy Self-Efficacy Scale (DALSS). The exploratory factor analysis results revealed that DALSS was found to have a 3-factor structure with 24 items. The dimensions were named as “Using appropriate tool and assessment type”, “Feedback and developing appropriate tools for the objectives and levels,” and “Preparing exams and evaluating the usability of the scores.” Confirmatory factor analysis (CFA) results showed that the hypothetical structure of the scale fit the data. Second-order CFA also confirmed the 3-factor structure of digital assessment literacy self-efficacy. Convergent and divergent validity results proved evidence for construct validity. Analyses based on known-group validity revealed that the DALSS is a discriminative instrument. The internal consistency reliability values of DALSS and its factors were found to vary between .96 and .98. The findings for practice revealed that although DALSS scores did not differentiate according to gender, professional experience and teaching level, they significantly differed in terms of teaching fields.

Keywords

• Assessment literacy
• Digital literacy
• Digital assessment
• Scale development
• Self-efficacy

References

1. American Federation of Teachers [AFT], National Council on Measurement in Education [NCME], & National Education Association [NEA] (1990). Standards for teacher competence in educational assessment of students. Educational Measurement: Issues and Practice, 9(4), 30-32.
2. Al-Bahlani, S.M. (2019). Assessment literacy: A study of EFL teachers' assessment knowledge, perspectives, and classroom behaviors [Doctoral dissertation, The University of Arizona].  Available from ProQuest Dissertations & Theses Global (2229389562). https://www.proquest.com/dissertations-theses/assessment-literacy-study-efl-teachers-knowledge/docview/2229389562/se-2
3. Al-Bahlani, S.M., & Ecke, P. (2023). Assessment competence and practices including digital assessment literacy of postsecondary English language teachers in Oman. Cogent Education, 10(2), 2239535. https://doi.org/10.1080/2331186X.2023.2239535
4. Anderson, J.C., & Gerbing, D.W. (1984). The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis. Psychometrika, 49(2), 155-173. https://doi.org/10.1007/BF02294170
5. Boateng, G.O., Neilands, T.B., Frongillo. E.A., Melgar-Quiñonez. H.R., & Young, S.L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Front Public Health. 11(6), 1 18. https://doi.org/10.3389/fpubh.2018.00149
6. Browne, M.W., & Cudeck, R. (1993). Alternative ways of assessing model fit. Sage Focus Editions, 154, 136-136. https://doi.org/10.1177/0049124192021002005
7. Cabezas-Gonzalez, M., Casillas-Martín, S., Sanches-Ferreira, M., & Teixeira Diogo, F.L. (2017). Do gender and age affect the level of digital competence? A study with university students. Fonseca Journal of Communication, 15, 109 125. https://doi.org/10.14201/fjc201715109125
8. Campbell, D.T., & Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81-105.

9. Cattell, R.B. (1978). The scientific use of factor analysis in behavioral and life sciences. Plenum Press. https://doi.org/10.1007/978-1-4684-2262-7
10. Chan, C.K.Y., & Luo, J. (2020). A four-dimensional conceptual framework for student assessment literacy in holistic competency development. Assessment & Evaluation in Higher Education, 46(3), 451-466. https://doi.org/10.1080/02602938.2020.1777388
11. Comrey, A.L., & Lee, H.B. (1992). A first course in factor analysis. Lawrence Erlbaum.
12. Cronbach, L.J., & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin 52(4), 281. https://doi.org/10.1037/h0040957
13. DeLuca, C., & Klinger, D.A. (2010). Assessment literacy development: Identifying gaps in teacher candidates’ learning. Assessment in Education: Principles, Policy & Practice, 17(4), 419-438. https://doi.org/10.1080/0969594X.2010.516643
14. DeLuca, C., LaPointe-McEwan, D., & Luhanga, U. (2016a). Approaches to classroom assessment inventory: A new instrument to support teacher assessment literacy. Educational Assessment, 21(4), 248-266. https://doi.org/10.1080/10627197.2016.1236677
15. DeLuca, C., LaPointe-McEwan, D., & Luhanga, U. (2016b). Teacher assessment literacy: A review of international standards and measures. Educational Assessment, Evaluation, and Accountability, 28, 251–272. https://doi.org/10.1007/s11092-015-9233-6
16. Edwards, F. (2016). A rubric to track the development of secondary pre-service and novice teachers’ summative assessment literacy. Assessment in Education: Principles, Policy & Practice, 24(2), 1-23. https://doi.org/10.1080/0969594X.2016.1245651
17. Erkuş, A. (2012). Psikolojide ölçme ve ölçek geliştirme [Measurement and scale development in Psychology]. Pegem Akademi.
18. Eyal, L. (2012). Digital assessment literacy - the core role of the teacher in a digital environment. Educational Technology & Society, 15(2), 37 49. https://www.jstor.org/stable/jeductechsoci.15.2.37
19. Fraenkel, J.R., Wallen, N.E., & Hyun, H.H. (2011). How to design and evaluate research in education. McGraw-Hill.
20. Fornell, C., & Larcker, D.F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18, 39–50.
21. Fulcher, G. (2012). Assessment literacy for the language classroom. Language Assessment Quarterly, 9, 113–132. https://doi.org/10.1080/15434303.2011.642041
22. Hair, J.F., Black, W.C., Babin, B.J., & Anderson, R.E. (2009). Multivariate data analysis. Prentice-Hall.
23. Hu, L.T., & Bentler, P.M. (1999). Cut-off criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. https://doi.org/10.1080/10705519909540118
24. Kline, R.B. (2016). Principle and practice of structural equation modeling. The Guilford Press.
25. Koyuncu, I., & Kılıç, A. (2019). The use of exploratory and confirmatory factor analyses: A document analysis. Education and Science, 44(198), 361-388.
26. Lawshe, C.H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563-575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x
27. Lucas, M., Bem-Haja, P., Siddiq, F., Moreira, A., & Redecker, C. (2021). The relation between in-service teachers’ digital competence and personal and contextual factors: What matters most? Computers & Education, 160, 104052. https://doi.org/10.1016/j.compedu.2020.104052.
28. Mertler, C.A. (2004). Secondary teachers' assessment literacy: Does classroom experience make a difference? American Secondary Education, 33(1), 49 64. https://www.jstor.org/stable/41064623
29. Mertler, C.A., & Campbell, C.S. (2005). Measuring teachers’ knowledge and application of classroom assessment concepts: Development of the assessment literacy inventory (Paper presentation). The annual meeting of the American Educational Research Association, Montreal, Quebec, Canada.
30. Ministry of National Education [MoNE] (2017). General qualifications for teaching profession. General Directorate of Teacher Training and Development. https://oygm.meb.gov.tr/dosyalar/StPrg/Ogretmenlik_Meslegi_Genel_Yeterlikleri.pdf
31. Mindrila, D. (2010). Maximum likelihood (ML) and diagonally weighted least squares (DWLS) estimation procedures: A comparison of estimation bias with ordinal and multivariate non-normal data. International Journal of Digital Society, 1(1), 60-66.
32. Morgado, F.F., Meireles, J.F., Neves, C.M., Amaral, A.C., & Ferreira, M.E. (2018). Scale development: ten main limitations and recommendations to improve future research practices. Psicologia: Reflexão e Crítica, 30(3), 1-20. https://doi.org/10.1186/s41155-016-0057-1
33. Nimehchisalem, V., & Bhatti, N. (2019). A review of literature on language assessment literacy in last two decades (1999-2018). International Journal of Innovation, Creativity and Change, 8(11), 44-59.
34. Nyagi, K., & Rajendran, M. (2020). Pre-service teachers’ approaches to classroom assessment. Humanities & Social Sciences Reviews, 8(1), 666 673. https://doi.org/10.18510/hssr.2020.8180
35. Plake, B.S., & Impara, J.C. (1992). Teacher competencies questionnaire description. University of Nebraska.
36. Popham, W.J. (2011). Assessment literacy overlooked: A teacher educator’s confession. The Teacher Educator, 46, 265–273. https://doi.org/10.1080/08878730.2011.605048
37. R Core Team (2023). R: A Language and environment for statistical computing. (Version 4.3) [Computer software]. https://cran.r-project.org. (R packages retrieved from CRAN snapshot 2024-01-09).
38. Rezai, A., Alibakhshi, G., Farokhipour, S., & Miri, M. (2021). A phenomenographic study on language assessment literacy: hearing from Iranian university teachers. Language Testing in Asia, 11(26), 1-25. https://doi.org/10.1186/s40468-021-00142-5
39. Rogier, D. (2014). Assessment literacy: Building a base for better teaching and learning. English Teaching Forum, 52(3), 2 13. https://americanenglish.state.gov/files/ae/resource_files/etf_52_3_02-13.pdf
40. Rosseel, Y. (2019). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. https://doi.org/10.18637/jss.v048.i02
41. Shams, J.A., Iqbal, M.Z., & Iqbal, M.Z. (2018). Investigation of classroom assessment literacy of university teachers of Punjab. Pakistan Journal of Distance and Online Learning, IV(II), 103-118. https://files.eric.ed.gov/fulltext/EJ1267033.pdf
42. Stiggins, R. (1991). Assessment literacy. The Phi Delta Kappan, 72, 534–539. https://learnline.cdu.edu.au/commonunits/documents/Scaffolding%20and%20formative%20assessment.pdf
43. Stiggins, R.J. (1999). Evaluating classroom assessment training in teacher education programs. Educational Measurement: Issues and Practice, 18(1), 23 17. https://doi.org/10.1111/j.1745-3992.1999.tb00004.x
44. Tabachnick, B.G., & Fidell, L.S. (2012). Using multivariate statistics. Allyn & Bacon.
45. The jamovi project (2024). Jamovi (Version 2.5) [Computer Software]. https://www.jamovi.org
46. Ullman, J.B. (2012). Structural equation modeling. In B.G. Tabachnik & L.S. Fidell (Eds.), Using multivariate statistics (6th ed.). Pearson.
47. Volante, L., & Fazio, X. (2007). Exploring teacher candidates' assessment literacy: Implications for teacher education reform and professional development. Canadian Journal of Education, 30(3), 749-770. https://eric.ed.gov/?id=EJ780818
48. Walters, F.S. (2010). Cultivating assessment literacy: Standards evaluation through language-test specification reverse engineering. Language Assessment Quarterly, 7(4), 317-342. https://doi.org/10.1080/15434303.2010.516042
49. Worthington, R.L., & Whittaker, T.A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806-838. https://doi.org/10.1177/0011000006288127
50. Yamtim, V., & Wong-Wanich, S. (2014). A study of classroom assessment literacy of primary school teachers. Procedia Social and Behavioral Sciences, 116, 2998 3004. https://doi.org/10.1016/j.sbspro.2014.01.696
51. Yong, A.G., & Pearce, S. (2013). A beginner’s guide to factor analysis: Focusing on exploratory factor analysis. Tutorials in Quantitative Methods for Psychology, 9(2), 79 94. https://doi.org/10.20982/tqmp.09.2.p079
52. Yuan, K.H. (2009). Normal distribution based pseudo ML for missing data: With applications to mean and covariance structure analysis. Journal of Multivariate Analysis, 100(9), 1900-1918.
53. Zhang, Z., & Burry-Stock, J.A. (2003). Classroom assessment practices and teachers’ self-perceived assessment skills. Applied Measurement in Education, 16(4), 323-342. https://doi.org/10.1207/S15324818AME1604_4