External School Evaluation Feedback and School Self-Evaluation: What Feedback Is Provided?

Abstract

Concerns with educational quality have led to the implementation of external school evaluation (ESE), based on the premise that these processes can provide valuable information about schools and, consequently, create conditions for improvement. Improvement is based on the feedback, commonly in the form of an evaluation report, resulting from evaluations, describing the reality of each school, and providing clues and guidance for action and progress. Nonetheless, ESE still has a relatively weak impact on overall school improvement. With this in mind, this paper focuses on the potential of evaluation reports to promote improvement, aiming to answer the question: What kind of feedback on school self-evaluation (SSE) does ESE provide to schools? Focusing on the Portuguese case, the paper analyses the feedback regarding school self-evaluation provided in evaluation reports from the northern region of Portugal. The study concludes that the feedback provided in the reports is mainly descriptive and generic, referencing issues that apply to all schools rather than targeting issues specific to each school. This leads to the hypothesis that the vagueness of ESE feedback can explain the limited contribution external evaluations make towards SSE improvement in particular, and school improvement overall. The example of Portugal and self-evaluation can help bring to light where ESE processes are underperforming and require investment to achieve their goals.

Keywords

• external school evaluation
• school self-evaluation
• evaluation reports
• feedback
• improvement

Supporting Institution

Foundation for Science and Technology (FCT)

Project Number

CEECIND/04172/2017

References

1. Afonso, A. J. (2009). Nem tudo o que conta em educação é mensurável ou comparável. Crítica à accountability baseada em testes estandardizados e rankings escolares. Revista Lusófona de Educação, 13, 13–29.
2. Afonso, A. J. (2010). Políticas educativas e auto-avaliação da escola pública portuguesa: apontamentos de uma experiência. Estudos em Avaliação Educacional, 21(46), 343–362.
3. Altrichter, H. & Kemethofer, D. (2015). Does accountability pressure through school inspections promote school improvement? School Effectiveness and School Improvement, 26(1), 32–56. https://doi.org/10.1080/09243453.2014.927369
4. Amado, J., Costa A. P., & Crusoé, N. (2017). A técnica de análise de conteúdo. In J. Amado (Ed.), Manual de Investigação Qualitativa em Educação (pp. 303–354). Imprensa da Universidade de Coimbra.
5. Ball, S. J. (1998). Big policies/Small world: An introduction to international perspectives in education policy. Comparative Education, 34(2), 119–130. https://doi.org/10.1080/03050069828225
6. Ball, S. J. (2001). Diretrizes políticas globais e relações políticas locais em educação. Currículo sem Fronteiras, 1(2), 99–116.
7. Bardin, L. (2011). Análise de conteúdo. Edições 70.
8. Behnke, K. & Steins, G. (2017). Principals’ reactions to feedback received by school inspection: A longitudinal study. Journal of Educational Change, 18, 77–106. https://doi.org/10.1007/s10833-016-9275-7

9. Bidarra, M. G., Barreira, C. F., Valgôde, M. M., Vaz-Rebelo, M. P., & Alferes, V. R. (2018). Atitudes dos professores face à avaliação de escolas. Indagatio Didactica, 10(2), 227–248. https://doi.org/10.34624/id.v10i2.11337 Bowen, G. (2009). Document analysis as a qualitative research method. Qualitative Research Journal, 9(2), 27–40. http://dx.doi.org/10.3316/QRJ0902027
10. Brady, A. M. (2019). Anxiety of performativity and anxiety of performance: Self-evaluation as bad faith. Oxford Review of Education, 45(5), 605–618. https://doi.org/10.1080/03054985.2018.1556626
11. Brown, M., McNamara, G., Ohara, J., O’Brien, S., Faddar, J., & Young, C. (2018). Integrated co-professional evaluation? Converging approaches to school evaluation across frontiers. Australian Journal of Teacher Education, 43(12), 76–90. http://dx.doi.org/10.14221/ajte.2018v43n12.6
12. Coe, R. (2009). School improvement: Reality and illusion. British Journal of Educational Studies, 57(4), 363–379. https://doi.org/10.1111/j.1467-8527.2009.00444.x
13. Devos, G. & Verhoeven, J. (2003). School self-evaluation conditions and caveats: The case of secondary schools. Educational. Management, Administration & Leadership, 31(4), 403–420. https://doi.org/10.1177/0263211X030314005 Datnow, A. & Hubbard, L. (2016). Teacher capacity for and beliefs about data-driven decision making: a literature review of international research. Journal of Education Change, 17, pp. 7-28. https://doi.org/10.1007/s10833-015-9264-2
14. Ehren, M. & Swanborn, M. (2012). Strategic data use of schools in accountability systems. School Effectiveness and School Improvement, 23(2), 257-280. https://doi.org/10.1080/09243453.2011.652127
15. Ehren, M. & Visscher, A. (2008). The relationships between school inspections, school characteristics and school improvement. British Journal of Educational Studies, 56(2), 205–227. http://dx.doi.org/10.1111/j.1467-8527.2008.00400.x
16. Eurydice (2004). Evaluation of schools providing compulsory education in Europe. https://op.europa.eu/pt/publication-detail/-/publication/96a863b8-2eef-4853-981d-09552ced6f95
17. Eurydice (2015). Assuring quality in education: Policies and approaches to school evaluation in Europe. http://eacea.ec.europa.eu/education/eurydice/documents/thematic_reports/178EN.pdf
18. Faubert, V. (2009). School evaluation: Current practices in OECD countries and a literature review, OECD Education Working Papers, 42, OECD Publishing. http://dx.doi.org/10.1787/218816547156 Ferreira, C. (2016). Avaliação externa de escolas em Portugal: Reflexões sobre potencialidades e constrangimentos. Meta: Avaliação, 8(23), 359–379. Figueiredo, C., Leite, C., & Fernandes, P. (2016a). O desenvolvimento do currículo no contexto de uma avaliação de escolas centrada nos resultados: Que implicações?. Currículo sem Fronteiras, 16(3), 646-664.
19. Figueiredo, C., Leite, C., & Fernandes, P. (2016b). The curriculum in school external evaluation frameworks in Portugal and England. Research in Comparative & International Education, 11(3), 282-297. https://doi.org/10.1177/1745499916661933 Figueiredo, C., Leite, C., & Fernandes, P. (2017). Avaliação externa de escolas: Do discurso às práticas: Uma análise focada em Portugal e em Inglaterra. Meta: Avaliação, 25, 1-31. https://doi.org/10.22347/2175-2753v9i25.1205 Figueiredo, C., Leite, C., & Fernandes, P. (2018). Uma tipologia para a compreensão da avaliação de escolas. Revista Brasileira de Educação, 23, 25p. http://dx.doi.org/10.1590/s1413-24782018230018
20. Figueiredo, C. (2023). Autoavaliação de escolas: O quê? Como? Com quem? E depois?. Ensaio: avaliação e políticas públicas em educação, 31(120), 1-24. https://doi.org/10.1590/S0104-40362023003103545 Gaertner, H., Wurster, S., & Pant, H. (2014). The effect of school inspections on school improvement. School Effectiveness and School Improvement, 25(4), 489–508. https://doi.org/10.1080/09243453.2013.811089 García, B. (2013). El papel de la inspección escolar en la mejora de los resultados educativos. Educar, 49(1), 67–82.
21. Geel, M.; Visscher, A. & Teunis, B. (2017) School characteristics influencing the implementation of a data-based decision making intervention. School Effectiveness and School Improvement, 28(3), pp. 443-462. https://doi.org/10.1080/09243453.2017.1314972
22. Gustafsson, J. -E., Ehren, M. C. M., Conyngham, G., McNamara. G., Altrichter, H., & O’Hara, J. (2015). From inspection to quality: Ways in which school inspection influences change in schools. Studies in Educational Evaluation, 47, 47–57. http://dx.doi.org/10.1016/j.stueduc.2015.07.002
23. Gutwirth, G., Goffin, E., & Vanhoof, J. (2021). Sensemaking unraveled: How teachers process school performance feedback data. Studia Paedagogica, 26(4). https://doi.org/10.5817/SP2021-4-4
24. IGEC (2010). Folheto de divulgação da avaliação externa das escolas - 2010–2011. https://www.igec.mec.pt/upload/AEE_2011/AEE_2010_11_Folheto.pdf IGEC (2011). Avaliação externa das escolas: Avaliar para a melhoria e a confiança – 2006-2011. https://www.igec.mec.pt/upload/Relatorios/AEE_2006_2011_RELATORIO.pdf IGEC (2016). Quadro de referência para a avaliação externa das escolas. Avaliação externa das escolas 2016–2017. https://www.igec.mec.pt/upload/AEE_2016-2017/AEE_16_17_(1)_Quadro_de_Referencia.pdf IGEC (2018). Avaliação externa das escolas 2014–2015 a 2016–2017 — Relatório. https://www.igec.mec.pt/upload/Relatorios/AEE_2014-2017_RELATORIO.pdf
25. IGEC (2019). Âmbito, princípios e objetivos. Terceiro ciclo da avaliação externa das escolas. https://www.igec.mec.pt/upload/AEE3_2018/AEE_3_Amb_princ_objetivos.pdf
26. Justino, D. & Almeida, S. (2016). Inspecionar e avaliar: Origens da regulação coerciva da educação em Portugal. Educação, Sociedade e Culturas, 47, 31–51.
27. Leite, C. & Marinho, P. (2021). A figura do “amigo crítico” no desenvolvimento de culturas de autoavaliação e melhoria de escolas. Ensaio: Avaliação e Políticas Públicas em Educação, 110, 90–111. https://doi.org/10.1590/s0104-40362020002802615
28. Leite, C., Fernandes, P., & Rodrigues, L. (2020). Trajetórias para a institucionalização de uma cultura de autoavaliação nas escolas: entre possibilidades e limites. Revista de Estudos Curriculares, 11(1), 102–121.
29. Leite, C., Morgado, J. C., & Seabra, F. (2014). External school evaluation in Portugal: A glance at the impacts on curricular and pedagogical practices. European Journal of Curriculum Studies, 1(1), 33–43. Leite, C., Rodrigues L., & Fernandes, P. (2006). A auto-avaliação das escolas e a melhorias da qualidade da educação: Um olhar reflexivo a partir de uma situação. Revista de Estudos Curriculares, 4(1), 21–45.
30. Lindahl, R. & Beach, R. (2013). The role of evaluation in the school improvement process. Planning and Changing, 44(1/2), 56–72. Lingard, B. (2000). It is and it isn’t: Vernacular globalization, educational policy and restructuring. In Nicholas Burbules & Carlos Torres (Eds.), Globalization and Education: Critical Perspectives (pp. 79–108). New York: Routledge.
31. Lingard, B. (2011). Policy as numbers: Ac/counting for educational research. Australian Educational Researcher, 38(4), 355–382.
32. Macbeath, J. (1999). Schools must speak for themselves: The case of school self-evaluation. Routledge.
33. MacBeath, J. (2004). Putting the self back into self-evaluation. Improving Schools, 7(1), 87–91. https://doi.org/10.1177/1365480204043309
34. MacBeath J. (2008) Leading learning in the self-evaluating school. School Leadership and Management, 28(4), 385–399. https://doi.org/10.1080/13632430802292332
35. McNamara, G. & O’Hara, J. (2012). From looking at our schools (LAOS) to whole school evaluation–management, leadership and learning (WSE-MLL): The evolution of inspection in Irish schools over the past decade. Educational Assessment Evaluation and Accountability 24, 79–97. https://doi.org/10.1007/s11092-012-9143-9
36. McNamara, G., O’Hara, J., & Aingléis, B. N. (2002). Whole-school evaluation and development planning: An analysis of recent initiatives in Ireland. Educational Management & Administration, 30(2), 201–211. https://doi.org/10.1177/02611X02030002510
37. Nayir, K. F., & McNamara, G. (2014). The increasingly central role of school self-evaluation in inspection systems across Europe: The case of Ireland. Turkish Journal of Education, 3(1), 48–59.
38. Olafsdóttir, B., Jónasson, J., & Sigurðardóttir, A. (2022). Use and impact of external evaluation feedback in schools. Studies in Educational Evaluation, 74. https://doi.org/10.1016/j.stueduc.2022.101181
39. Osga, J. & Lawn, M. (2014). Frameworks of regulation: Evidence, knowledge and judgment in inspection. Sisyphus. Journal of Education, 1(II), 7–15.
40. Penninckx, M. & Vanhoof, J. (2015). Insights gained by schools and emotional consequences of school inspections. A review of evidence. School Leadership & Management, 35(5), 477–501. https://doi.org/10.1080/13632434.2015.1107036 Penninckx, M., Vanhoof, J., De Maeyer, S., & Petegem, P. (2014) Exploring and explaining the effects of being inspected. Educational Studies, 40(4), 456–472. https://doi.org/10.1080/03055698.2014.930343
41. Penninckx, M., Vanhoof, J., De Maeyer, S., & Petegem, P. (2016) Explaining effects and side effects of school inspections: A path analysis. School Effectiveness and School Improvement, 27(3), 333-347. https://doi.org/10.1080/09243453.2015.1085421
42. Petegem, P. & Vanhoof, J. (2007). Towards a model of effective school feedback: School heads’ points of view. Educational Research and Evaluation, 13(4), 311–325. https://doi.org/10.1080/13803610701702522
43. Portugal (1986), “Basic Law of the Portuguese Education Systems, law no. 46/86”, available at https://dre.pt/web/guest/pesquisa/-/search/222418/details/normal?p_p_auth=D688OvBC.
44. Portugal (2002), “Law no. 31/2002”, available at https://diariodarepublica.pt/dr/detalhe/lei/31-2002-405486. Quintelier, A., Vanhoof, J., & De Maeyer, S. (2018) Understanding the influence of teachers’ cognitive and affective responses upon school inspection feedback acceptance. Educational Assessment, Evaluation and Accountability, 30, 399–431. https://doi.org/10.1007/s11092-018-9286-4
45. Quintelier, A., De Maeyer, S., & Vanhoof, J. (2020) The role of feedback acceptance and gaining awareness on teachers’ willingness to use inspection feedback. Educational Assessment, Evaluation and Accountability, 32, 311–333. https://doi.org/10.1007/s11092-020-09325-9 Richards, C. (2004). Towards respectful inspection – a critique of proposed arrangements for school inspection and self-evaluation. Education Review, 18(1), 42–48.
46. Richards, C. (2020) Viewpoint: Thoughts on whole school inspection post-Covid. London Review of Education, 18(3), 511–18. https://doi.org/10.14324/LRE.18.3.13
47. Sá, V. (2018). Avaliação institucional de escolas de Educação Básica em Portugal: políticas, processos e práticas. Ensaio: Avaliação e Políticas Públicas em Educação, 26(100), 801-821. https://doi.org/10.1590/S0104-40362018002601163
48. Sampaio, M., Figueiredo, C., Leite, C., & Fernandes, P. (2016). Efeitos da avaliação externa de escolas nos processos de autoavaliação: Convergências e tendências de ação. In C. Barreira, M. G. Bidarra & M. P. Vaz-Rebelo (Eds.). Estudos sobre avaliação externa de escolas (pp. 37–54). Porto Editora.
49. Sampaio, M. & Leite, C. (2016). A avaliação externa das escolas e os TEIP na sua relação com a justiça social. Educação, Sociedade & Culturas, 47, 115–136.
50. Schildkamp, K. (2019) Data-based decision-making for school improvement: Research insights and gaps, Educational Research, 61(3), 257–273. https://doi.org/10.1080/00131881.2019.1625716
51. Schildkamp, K., Vanhoof, J., Van Petegem, P., & Visscher, A. (2012). The use of school self-evaluation results in the Netherlands and Flanders. British Educational Research Journal, 38(1), 125–152. https://doi.org/10.1080/01411926.2010.528556
52. Seabra, F., Henriques, S., Mouraz, A., Abelha. M., & Tavares, A. (2022). Schools’ strengths and areas for improvement: Perspectives from external evaluation reports. Frontiers in Education. 7(868481). https://doi.org/10.3389/feduc.2022.868481
53. Sousa, A. & Terrasêca, M. (2015). Processos de autoavaliação de escolas ou avaliação interna do sistema?. Education Policy Analysis Archives, 23, 18p. https://doi.org/10.14507/epaa.v23.2085
54. Terrasêca, M. (2016). Autoavaliação, avaliação externa… afinal para que serve a avaliação das escolas?. Cad. Cedes, 36(99), 155–174. https://doi.org/10.1590/CC0101-32622016160581
55. Vanhoof, J. & Van Petegem, P. (2007). Matching internal and external evaluation in an era of accountability and school development: Lessons from a Flemish perspective. Studies in educational evaluation, 33(2), 101–119. https://doi.org/10.1016/j.stueduc.2007.04.001
56. Veloso, L., Abrantes, P., & Craveiro, D. (2011). A avaliação externa das escolas como processo social. Educação, Sociedade & Culturas, 33, 69–88.
57. Verhaeghe, G., Schildkamp, K., Luyten, H., & Valcke, M. (2015). Diversity in school performance feedback systems. School Effectiveness and School Improvement, 26(4), 612–638. https://doi.org/10.1080/09243453.2015.1017506
58. Visscher, A. & Coe, R. (2003). School performance feedback systems: Conceptualisation, analysis, and reflection, School Effectiveness and School Improvement, 14(3), 321–349. https://doi.org/10.1076/sesi.14.3.321.15842