A Meta-Analysis Study on Data Literacy Education For School Administrators and Teachers

Abstract

This meta-analysis study aimed to examine the effect of data literacy education, which affects data-based decision processes, on data use knowledge and skills of school administrators and teachers. Therefore, theses of data literacy education for school administrators and teachers and relevant studies in peer-reviewed journals were examined through several databases. The study was conducted using the Comprehensive Meta-Analysis (CMA) software, using a total of eight studies published between 2006-2021. The results revealed that the selected studies were heterogeneous. Therefore, a random effects model was applied in the study. The overall effect size value of data literacy education was calculated as 2.16, suggesting that a data literacy education makes a positive high contribution to data use knowledge and skills of school administrators and teachers. The subgroup analyzes conducted to determine the source of heterogeneity in results have shown that data literacy education did not differ by type and country of publications, but varied by type of participants, where studies conducted with mixed participants had high effect values. Based on these results, it can be suggested that data literacy education given to school administrators and teachers should be expanded in all countries and education levels, and a meta-analysis of studies conducted in correlation between data literacy and different variables.

Keywords

• Data literacy
• professional development
• school administrator
• teacher
• meta-analysis.

References

1. Sources marked with an asterisk (*) indicate studies included in the meta-analysis.
2. *Abrams, L. M., Varier, D., & Mehdi, T. (2021). The intersection of school context and teachers’ data use practice: Implications for an integrated approach to capacity building. Studies in Educational Evaluation, 69, 1-13. https://doi.org/10.1016/j.stueduc.2020.100868
3. Açıkel, C. (2009). Meta-analysis and its place in evidence based medicine. Bulletin of Clinical Psychopharmacology,19(2), 164-172.
4. Albrect, N., Lou, M., & Neill, S. (2014). The importance of data literacy in leadership education: Factors related to data use in instructional leadership. 03 June 2014, 10th International Academic Conference, Vienna.
5. Athanases, S., Wahleithner, J., & Bennett, L. (2012). Learning to attend to culturally and linguistically diverse learners through teacher inquiry in teacher education. Teachers College Record, 114(7), 1-50.
6. Barutçugil, İ. (2002). Bilgi yönetimi. Kariyer Publishing
7. Başol, G., & Johanson, G. (2009). Effectiveness of frequent testing over achievement: A meta-analysis study. International Journal of Human Sciences, 6(2), 99-121.
8. Bernhardt, V. L. (2009). Data use: Data-driven decision making takes a big-picture view of the needs of teachers and students. Journal of Staff Development, 30(1), 24-27.

9. *Bettesworth, L. R. (2006). Administrators' use of data to guide decision-making. [Doctoral dissertation]. University of Oregon, Eugene, OR. Borenstein, M., Hedges, L.V., Higgins, J.P.T., & Rothstein H. (2013). Comprehensive meta-analysis. Englewood, NJ: Biostat.
10. Carey, M., Grainger, P., & Christie, M. (2018). Preparing preservice teachers to be data literate: a queensland case study. Asia-Pacific Journal of Teacher Education, 46(3), 267-278. https://doi.org/10.1080/1359866X.2017.1402860
11. Childress, M. (2009). Data-driven decision making: The development and validation of an instrument to measure principals’ practices. Academic Leadership: The Online Journal, 7(1), 67-75.
12. Coburn, C. E., & Turner, E. O. (2011). Research on data use: A framework and analysis. Measurement: Interdisciplinary Research & Perspective, 9(4), 173–206. https://doi.org/10.1080/15366367.2011.626729
13. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. New York: Academic Pres.,
14. Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis. (2nd edition). New York: Russell Sage Publication.
15. Cowie, B., & Cooper, B. (2017). Exploring the challenge of developing student teacher data literacy. Assessment in Education: Principles, Policy and Practice, 24(2), 147-163. https://doi.org/10.1080/0969594X.2016.1225668
16. Cumming, G. (2012). Understanding the new statistics. New York: Routledge, Taylor and Francis Group.
17. Çarkungöz, E., & Ediz, B. (2009). Meta-analysis. Uludag University Journal of Faculty Veterinary Medicine, 28(1), 33-37.
18. Darling-Hammond, L., & Orphanos, S. (2006). Leadership development in California. Retrieved from https://cepa.stanford.edu/sites/default/files/12-Darling-Hammond%283-07%29.pdf
19. Datnow, A., Park, V., & Wohlstetter, P. (2007). Achieving with data: How highperforming school systems use data to improve instruction for elementary students. Los Angeles, CA: Center on Educational Governance, Rossier School of Education, University of Southern California.
20. Dejear, M. L. (2016). A study of how culture, collaboration, and advocacy influence data-driven decision making at community colleges. [Doctor of philosophy]. Iowa State University, Ames, Iowa.
21. DeLuca, C., & Bellara, A. (2013). The current state of assessment education aligning policy, standards, and teacher education curriculum. Journal of Teacher Education, 64(4), 356-372. https://doi.org/10.1177/0022487113488144
22. Doğan, E. (2021). Evaluating the data-driven decision making process in school management according to the views of the administrators. [Unpublished doctoral dissertation]. Gazi University, Ankara.
23. Dunn, K., Airola, D., Lo, W., & Garrison, M. (2013). What teachers think about what they can do with data: Development and validation of the data driven decision-making efficacy and anxiety inventory. Contemporary Educational Psychology, 38(1), 87-98. http://dx.doi.org/10.1016/j.cedpsych.2012.11.002
24. Earl, L., & Katz, S. (2002). Leading schools in a data rich world. In K. Leithwood and P. Hallinger (Eds.), Second international handbook of educational leadership and administration. Kluwer.
25. *Ebbeler, J., Poortman, C., Schildkamp, K., & Pieters, J. (2016). The effects of a data use intervention on educators’ satisfaction and data literacy. Educational Assessment, Evaluation and Accountability, 29(1), 83-105. https://link.springer.com/article/10.1007/s11092-016-9251-z
26. Edwards, J., Lyons, C., & Jost, M. L. (1997). Quality instruction based on data to better meet student needs. Paper presented at the Annual Meeting of the American Educational Research Association (March 24-28). ERIC Document Reproduction NO. ED440119.
27. Ellis, P. D. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge: Cambridge University Press.
28. Faber, J. M., & Visscher, A. J. (2014). Digitale leerlingvolgsystemen: Een review van de effecten op leerprestaties. 4W. Retrieved from http://www.kennisnet.nl/onder zoek/alle-onderzoeken/digitale-leerlingvolgsystemen-een-review-van-deeffecten-op-leerprestaties/
29. Farley-Ripple, E., & Buttram, J. (2014). Developing collaborative data use through professional learning communities: Early lessons from Delaware. Studies in Educational Evaluation, 42, 41–53. http://dx.doi.org/10.1016%2Fj.stueduc.2013.09.006
30. Feldman, J. & Tung, R. (2001). Whole school reform: how schools use the data-based inquiry and decision-making process. Seattle, WA: American Educational Research Association.
31. Gambell, T. (2004). Teachers working around large-scale assessment: Reconstructing professionalism and professional development. English Teaching: Practice and Critique, 3(2), 48-73.
32. Gesel, S. A., LeJeune, L. M., Chow, J. C., Sinclair, A. C., & Lemons, C. J. (2020). A meta-analysis of the impact of professional development on teachers’ knowledge, skill, and self-efficacy in data-based decision-making. Journal of Learning Disabilities, 1–15. https://doi.org/10.1177/0022219420970196
33. *Green, J. L., Schmitt-Wilson, S., Versland, T., Kelting-Gibson, L., & Nollmeyer, G. E. (2016). Teachers and data literacy: A blueprint for professional development to foster data driven decision making. Journal of Continuing Education and Professional Development, 3(1), 14-32. http://dx.doi.org/10.7726/jcepd.2016.1002
34. Gummer, E., & Mandinach, E. (2015). Building a conceptual framework for data literacy. Teachers College Record, 117(4), 1-22.
35. Harris, N. (2011). The impact of professional development in data-based decision making on the teaching practices of educators. [Doctoral dissertation]. Walden University College of Education, Walden.
36. Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA, US: Sage Publications, Inc.
37. Jacobs, J., Gregory, A., Hoppey, D., & Yendol-Hoppey, D. (2009). Data literacy: Understanding teachers' data use in a context of accountability and response to intervention. Action in Teacher Education, 31(3), 41-55. https://doi.org/10.1080/01626620.2009.10463527
38. *Jimenez, B. A., Mims, P. J., & Browder, D. M. (2012). Data-based decisions guidelines for teachers of students with severe ıntellectual and developmental disabilities. Education and Training in Autism and Developmental Disabilities, 47(4), 407–413.
39. Johnson, T. (2015). Professional development effects on teachers’ perceptions in analyzing and using student data. [Doctoral dissertation]. Walden University College of Education, USA.
40. Jung, P. G., McMaster, K. L., Kunkel, A., Shin, J., & Stecker, P. M. (2018). Effects of data-based individualization for students with intensive learning needs: A meta-analysis. Learning Disabilities Research & Practice, 33(3), 144–155. https://doi.org/gd4g8m
41. Kerr, K. A., Marsh, J. A., Ikemoto, G. S., Darilek, H., & Barney, H. (2006). Strategies to promote data use for instructional improvement: actions, outcomes, and lessons from three urban districts. American Journal of Education, 112(4), 496-520.
42. Keuning, T., Van Gell, M., & Visscher, A. (2017). Why a Data-Based Decision-Making Intervention Works in Some Schools and Not in Others. Learning Disabilities Research & Practice, 32(1), 1–14. http://dx.doi.org/10.1111/ldrp.12124
43. Killion, J., & Bellamy, G. T. (2000). On the job: Data analysts focus school improvement efforts. Journal of Staff Development, 21(1), 27-31. Khan, H. R., Kim, J., & Chang, C. H. (2018). Toward an understanding of data literacy. http://hdl.handle.net/2142/100243
44. *Kippers, W. B., Poortman, C. L., Schildkamp, K., & Visscher, A. J. (2018). Data literacy: What do educators learn and struggle with during a data use intervention? Studies in Educational Evaluation, 56, 31-21. https://doi.org/10.1016/j.stueduc.2017.11.001
45. LaPointe-McEwan, D., DeLuca, C., & Klinger, D. (2017). Supporting evidence use in networked professional learning: the role of the middle leader. Educational Research, 59(2), 136-153. https://doi.org/10.1080/00131881.2017.1304346
46. Lange, C., Range, B., & Welsh, K. (2012). Conditions for effective data use to improve schools: Recommendations for school leaders. International Journal of Educational Leadership Preparation, 7(3), 1-11.
47. Levin, J. A., & Datnow, A. (2012). The principal role in data driven decision making: Using case study data to develop multimediator models of educational reform. School Effectiveness and School Improvemenet, 23(2), 179-201. http://dx.doi.org/10.1080/09243453.2011.599394
48. Luo, M., Albrecht, N., & Neill, S. (2015). Factors related to data use in instructional leadership: The importance of data literacy in leadership education. International Journal of Teaching and Education, 3(1), 24-44.
49. Mandinach, E. B., & Gummer, E. S. (2013). A systemic view of implementing data literacy in educator preparation. Educational Researcher, 42(1), 30-37. http://dx.doi.org/10.3102/0013189X12459803
50. Mandinach, E. B., & Gummer, E. S. (2016). What does it mean for teachers to be data literate: Laying out the skills, knowledge, and dispositions. Teaching and Teacher Education, 60. http://dx.doi.org/10.1016/j.tate.2016.07.011
51. Mandincah, E.B., Honey, M., & Light, D. (2006). A theoretical framework for datadriven decision making. Paper presented at the Annual Meeting of AERA, CA.
52. Marsh, J. A., Pane, J. F., & Hamilton, L. S. (2006). Santa Monica, CA: https://www.rand.org/content/dam/rand/pubs/occasional_papers/2006/RAND_OP170.
53. Mason, S. A. (2003). Learning from data: The role ofprofessional learning communities. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
54. McCray, M. (2014). Data driven decision-making and principals’ perceptions. [Doctoral dissertation]. Mississippi State University Department of Leadership and Foundations, Mississippi.
55. Means, B., Padilla, C., DeBarger, A., & Bakia, M. (2009). Implementing data-informed decision making in schools: Teacher access, supports and use. U.S. Department of Education.
56. Mertler, C. (2004). Secondary teachers‘ assessment literacy: Does classroom experience make a difference? American Secondary Education, 33(1), 49-64.
57. Niemeyer, K. D. (2012). Data driven decision making in schools: The influence of professional development on educator perceptions. [Doctoral dissertation]. The University of Memphis.
58. Park, V. (2008). Beyond the numbers chase: How urban high school teachers make sense of data use. [Doctoral dissertation]. University of Southern California. Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences. MABlackwell Publishers Ltd.
59. Piro, J., Dunlap, K., & Shutt, T. (2014). A collaborative data chat: teaching summative assessment data use in pre-service teacher education. Cogent Education, 1(1), 1-24. https://doi.org/10.1080/2331186X.2014.968409
60. Piro, J., & Hutchinson, C. (2014). Using a data chat to teach instructional interventions: student perceptions of data literacy in an assessment course. The New Educator, 10(2), 95-111. http://dx.doi.org/10.1080/1547688X.2014.898479
61. Reeves, T. D., & Chiang, J. L. (2019). Effects of an asynchronous online data literacy intervention on pre-service and in-service educators’ beliefs, self-efficacy, and practices. Computers & Education, 136, 13-33. https://doi.org/10.1016/j.compedu.2019.03.004
62. Reeves, T. D., & Honig, S. L. (2015). A classroom data literacy intervention for pre-service teachers. Teaching and Teacher Education, 50(1), 90-101. http://dx.doi.org/10.1016/j.tate.2015.05.007
63. Rogers, M. A. (2015). A developmental study examining the value, effectiveness, and quality of a data literacy intervention. [Doctoral dissertation]. University of Iowa, Iowa.
64. *Rotondi, M. (2017). Teacher perception of data-driven instruction and collective decision making. [Master thesis]. California State University, Fresno.
65. Schildkamp, K., & Kuiper, W. (2010). Data-informed curriculum reform: Which data, what purposes, and promoting and hindering factors. Teaching and Teacher Education, 26(3), 482-496. http://dx.doi.org/10.1016/j.tate.2009.06.007
66. Schildkamp, K., & Lai, M. K. (2013). Data-based decision making: Conclusions and a data use framework. In K. Schildkamp, M. K. Lai, & L. Earl (Eds.), Data-based decision making in education: Challenges and opportunities (pp. 177–191). Dordrecht, The Netherlands: Springer.
67. Stecker, P. M., Fuchs, L. S., & Fuchs, D. (2005). Using curriculum-based measurement to improve student achievement: Review of research. Psychology in the Schools, 42(8), 795–819. https://doi.org/10.1002/pits.20113
68. Symonds, K. W. (2003). After the test: How schools are using data to close the achievement gap. San Francisco: Bay Area School Reform Collaborative.
69. Uiterwijk-Luijk, L., Krüuger, M., Zijlstra, B., & Volman, M. (2017). Inquiry-based leadership: the influence of affective attitude, experienced social pressure and self-efficacy. Journal of Educational Administration, 55(5), 492-509. https://doi.org/10.1108/JEA-12-2015-0114
70. Van Geel, M., Keuning, T., Visscher, A. J., & Fox, J. P. (2016). Assessing the effects of a school-wide databased decision-making intervention on student achievement growth in primary schools. American Educational Research Journal, 53(2), 360–394. https://doi.org/10.3102/0002831216637346
71. *Van Geel, M., Keuning, T., Visscher, A. J., & Fox, J. P. (2017). Changes in educators' data literacy during a data-based decision-making intervention. Teaching and Teacher Education, 64, 187-198. http://dx.doi.org/10.1016/j.tate.2017.02.015
72. Vanhoof, J., Verhaegheb, G., Jean Pierre Verhaegheb, J. P., Valckeb, M., & Petegama, P. V. (2011). The influence of competences and support on school performance feedback use. Educational Studies, 37(2), 141–154. https://doi.org/10.1080/03055698.2010.482771
73. Van Kuijk, M. F., Deunk, M. I., Bosker, R. J., & Ritzema, E. S. (2016). Goals, data use, and instruction: The effect of a teacher professional development program on reading achievement. School Effectiveness and School Improvement, 27(2), 1–22. https://doi.org/10.1080/09243453.2015.1026268
74. Verbiest, E., Pol, M., Vanlommel, K., Mahieu, P., Lazarova, B., Malmberg, K., Neimane, S., Erculj, J., Savarin, A., Gleizups, A., & Hortlund, T. (2014). Becoming a data-wise school leader: developing leadership capacity for data-informed school improvement. Journal of Contemporary Educational Studies, (4), 64-82.
75. Wayman, J. (2005). Involving teachers in data-driven decision making: using computer data systems to support teacher inquiry and reflection. Journal of Education for Students Placed at Risk, 10(3), 295-308. http://dx.doi.org/10.1207/s15327671espr1003_5
76. Wayman, J. C., Cho, V., Jimerson, J. B., & Spikes, D. D. (2012). District-wide effects on data use in the classroom. Educational Policy Analysis Archives, 20(25). http://dx.doi.org/10.14507/epaa.v20n25.2012
77. Wayman, J. C., & Jimerson, J. B. (2013). Teacher needs for data-related Professional learning. Studies in Educational Evaluation, 42, 25-34. https://doi.org/10.1016/j.stueduc.2013.11.001
78. Williams, D., & Coles, L. (2007). Teachers’ approaches to finding and using research evidence: an information literacy perspective. Educational Research, 49(2), 185-206. https://doi.org/10.1080/00131880701369719
79. Wolf, F.M. (1988). Meta-analysis quantitative methods for research synthesis. Sage Publications.
80. Young, V. M., & Kim, D. H. (2010). Using assessments for instructional improvement: a literature review. Education Policy Analysis Archives, 18(19).