Redefining the impact of professional development in education with ProDES (Professional Development Evaluation Scale)

Year 2024, , 733 - 757, 15.11.2024

Mustafa Özgenel , Martin Brown , Joe O Hara , Metin Özkan

https://doi.org/10.21449/ijate.1327238

Abstract

This study introduces the Professional Development Evaluation Scale (ProDES), a tool that has been developed to evaluate the impact of professional development as it relates to participants' Learning and Use of New Knowledge and Skills, Organisational Support, Student Learning Outcomes, and reactions. Grounded in Guskey’s (2000) framework for evaluating Professional Development, ProDES was developed with data from five study groups in Turkey and underwent refinement across four factors. Exploratory and Confirmatory Factor Analyses confirmed the scale's structure, accounting for 62.72% of the total variance, with robust fit indices. Within this, ProDES demonstrated high internal consistency and test-retest reliability, with significant correlations validating its effectiveness. The scale's high internal consistency and test-retest reliability ensure that it can be used to make evidence-informed decisions that can foster more effective and supportive professional development activities. As a result, by identifying which professional development initiatives lead to improvements, those associated with professional development can use resources more efficiently, leading to enhanced school and system-wide improvements. Moreover, the use of ProDES can also help schools and education systems track progress over time, making ProDES an invaluable tool for continuous improvement and strategic planning.

Keywords

Professional development, Assessment of professional development, Scale development, Professional learning

Ethical Statement

İstanbul Sabahattin Zaim Üniversitesi, 30.05.2023-E.54478.

Supporting Institution

TÜBİTAK

Project Number

1059B192000757

Thanks

Akif Avcu, Ersin Yağan, Fazilet Taşdemir, Musa Bardak, Gamze Kasalak, Hikmet Şevgin, Hilmi Koç, İsa Bahat, Metin Işık, Pınar Özkan, Polat Erdoğan, Sedat Gümüş, Seçil Sümbül, Şebnem Yazıcı and Yusuf Cerit who expressed their opinions as experts in the scale development process and and to all our colleagues who supported the data collection process we thank you.

References

• Adey, P. (2004). The professional development of teachers: Practice and theory. Springer Science & Business Media.
• Altunışık, R., Coşkun, R., Bayraktaroğlu, S., & Yıldırım, E. (2004). Sosyal bilimlerde araştırma yöntemleri [Research methods in social sciences]. Sakarya Kitabevi.
• Assor, A., & Oplatka, I. (2003). Towards a comprehensive conceptual framework for understanding principals’ personal-professional growth. Journal of Educational Administration, 41(5), 471-497.
• Baker, F.B. (2017). Madde tepki kuramının temelleri [Fundamentals of item response theory] (N. Güler, Çev., Ed.). Pegem.
• Barth, R.S. (1986). Principal centered professional development. Theory Into Practice, 25(3), 156-160.
• Baykul, Y. (2015). Eğitim ve psikolojide ölçme: Klasik test teori ve uygulaması [Measurement in education and psychology: Classical test theory and practice]. ÖSYM.
• Bentler, P.M. & Bonett, D.G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88(3), 588-606.
• Best, J.W. & Kahn, J.V. (2017). Eğitimde araştırma yöntemleri [Research methods in education] (O. Köksal. Çev. Ed.). Eğitimevi.
• Blandford, S. (2004). Professional development manual: A practical guide to planning and evaluating successful staff development. Pearson Education.
• Blandford, S. (2012). Managing professional development in schools. Routledge.
• Bredeson, P.V. (2000) The school principal's role in teacher Professional development. Journal of In-Service Education, 26(2), 385-401.
• Brown, T.A. & Moore, M.T. (2013). Confirmatory factor analysis. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 361-379). Guilford Press.
• Büyüköztürk, Ş. (2012). Sosyal bilimler için veri analizi el kitabı [Handbook of data analysis for the social sciences]. Pegem.
• Büyüköztürk, Ş. (2002). Faktör analizi: Temel kavramlar ve ölçek geliştirmede kullanımı [Factor analysis: Basic concepts and use in scale development]. Kuram ve Uygulamada Eğitim Yönetimi, 32, 470-483.
• Byrne, B.M. (2012). Structural equation modeling with Mplus: Basic concepts, applications, and programming. Routledge Taylor & Francis Group.
• Campbell, A., McNamara, O., & Gilroy, P. (2004). Practitioner research and professional development in education. Sage.
• Cohen, R.J. & Swerdlik, M.E. (2015). Psikolojik test ve değerlendirme [Psychological testing and assessment] (E. Tavşancıl, Çev. Ed.). Nobel.
• Cohen, S. (2004). Teachers' professional development and the elementary mathematics classroom: Bringing understandings to light. Routledge.
• Cole, P. (2008). Leadership and professional learning: Forty actions leaders can take to ımprove Professional learning. IARTV.
• Çakır, Ö., & Horzum, M.B. (2014). Adaptation motivation toward web-based professional development scale and examining pre-service teachers’ motivation toward web-based professional development perception in terms of different variables. Procedia-Social and Behavioral Sciences, 131, 144-148.
• Çokluk, Ö., Şekerçioğlu, G., & Büyüköztürk, Ş. (2012). Sosyal bilimler için çok değişkenli istatistik SPSS ve Lisrel uygulamaları [Multivariate statistics for social sciences: SPSS and Lisrel applications]. Pegem.
• Day, C. (2002). Developing teachers: The challenges of lifelong learning. Routledge.
• Desimone, L. (2011). A Primer on professional development. Phi Delta Kappan, 92(6), 68-71.
• DeVellis, R.F. (2014). Ölçek geliştirme kuram ve uygulamalar [Scale development: theory and applications] (T. Totan, Çev. Ed.). Nobel
• Diaz-Maggioli, G. (2004). Teacher-centered professional development. ASCD.
• Dijkstra, E.M. (2009). Hoe professioneel is de hedendaagse onderwijsprofessional? (What is the professionality of the contemporary educational professional?) [Unpublished master thesis]. Rijksuniversiteit Groningen.
• Easton, L.B. (2008). From professional development to professional learning. Phi Delta Kappan, 89(10), 755-761.
• Erkuş, A. (2014). Psikolojide ölçme ve ölçek geliştirme I: Temel kavramlar ve işlemler [Measurement and scale development in psychology I: Basic concepts and procedures]. Pegem.
• Eroğlu, M., & Özbek, R. (2018). Development of professional development activities scale for teachers. Journal of Current Researches on Social Sciences, 8(3), 185-208.
• Eroğlu, M. (2019). Öğretmenlerin mesleki gelişime katılımlarıyla, mesleki gelişime yönelik tutumları, kendi kendine öğrenmeye hazır bulunuşlukları ve destekleyici okul özellikleri arasındaki ilişkinin incelenme [Investigatıon of the relationship between teachers 'participation in professional development and the attitudes toward professıonal development, readıness for selfdirected learning and supportıve school characteristics] [Unpublished Doctoral dissertation]. İnönü Üniversitesi, Malatya.
• Eroğlu, M., & Özbek, R. (2020). Mesleki gelişim etkinlikleri ölçeğinin uyarlanması: Geçerlik ve güvenirlik çalışması [Adaptation of professional development activities scale: Validity and reliability study]. Turkish Studies, 15(4), 2611-2628.
• Field, A. (2009). Discovering statistics using SPSS. Sage.
• Fullan, M. (1994). Change forces: Probing the depths of educational reform. Palmer Press.
• Garet, M.S., Porter, A.C., Desimone, L., Birman, B.F., & Yoon, K.S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915-945.
• George, D., & Mallery, P. (2016). IBMM SPSS statistics step by step. Routledge.
• Guskey, T.R. (1986). Staff development and the process of teacher change. Educational Researcher, 15(5), 5-12.
• Guskey, T.R. (1999). New perspectives on evaluating professional development. Paper presented at the Annual Meeting of the American Educational Research Association, Montreal, Quebec, Canada.
• Guskey, T.R. (2000). Evaluating professional development. Corwin press.
• Guskey, T.R. (2002a). Professional development and teacher change. Teachers and Teaching: Theory and Practice, 8(3/4), 381-391.
• Guskey, T.R. (2002b). Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45-51.
• Guskey, T.R. (2003a). The characteristics of effective professional development: A synthesis of lists. Paper presented at the Annual Meeting of the American Educational Research Association. Chicago, IL, April 21-25, 2003).
• Guskey, T.R. (2003b). Analyzing lists of the characteristics of effective professional development to promote visionary leadership. NASSP Bulletin, 87(637), 4-20.
• Guskey, T.R., & Yoon, K.S. (2009). What works in professional development? Phi Delta Kappan, 90(7), 495-500.
• Gümüş, S., Apaydın, Ç., & Bellibaş, M.Ş. (2018). Öğretmen mesleki öğrenme ölçeğinin Türkçeye uyarlanması: Geçerlik ve güvenirlik çalışması [Adaptation of teacher professional learning scale to Turkish: The validity and reliability study]. Eğitim ve İnsani Bilimler Dergisi: Teori ve Uygulama, 9(17), 107-124.
• Hair, J.F., Jr., Anderson, R.E., Tatham, R.L., & Black, W.C. (1998). Multivariate data analysis (5th ed.). Pearson Education Inc.
• Hallinger, P. (2003). Leading educational change: Reflections on the practice of instructional and transformational Leadership. Cambridge Journal of Education, 33(3), 329-352.
• Helm, J.L., Castro-Schilo, L., & Oravecz, Z. (2017). Bayesian versus maximum likelihood estimation of multitrait–multimethod confirmatory factor models. Structural Equation Modeling: A Multidisciplinary Journal, 24(1), 17-30.
• Hinkin (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 2 (1), 104-121.
• Hinkin, T.R., Tracey, J.B., & Enz, C.A. (1997). Scale construction: Developing reliable and valid measurement instruments. Journal of Hospitality & Tourism Research, 21(1), 100-120.
• Ho, R. (1998). Handbook of univariate and multivariate data analysis and interpretation with SPSS. Chapman & Hall/CRC, Boca Raton.
• Hof, M.W. (2012). Questionnaire evaluation with factor analysis and Cronbach’ s Alpha: An example. Retrieved from http://www.let.rug.nl/nerbonne/teach/rema-stats-meth-seminar/student-papers/MHof-QuestionnaireEvaluation-2012-Cronbach-FactAnalysis.pdf
• Howard, M.C. (2016). A review of exploratory factor analysis decisions and overview of current practices: What we are doing and how can we improve? International Journal of Human- Computer Interaction, 32(1), 51–62.
• Hu, L., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55.
• Jöreskog, K.G. (2004). On chi-squares for the ındependence model and fit measures in Lisrel. http://www.ssicentral.com/lisrel/techdocs/ftb.pdf
• Kalaycı, Ş. (2010). SPSS uygulamalı çok değişkenli istatistik teknikleri [Multivariate statistical techniques with SPSS applications]. Asil.
• Kao, C.P., Wu, Y.T., & Tsai, C.C. (2011). Elementary school teachers’ motivation toward web-based professional development, and the relationship with Internet self-efficacy and belief about web-based learning. Teaching and Teacher Education, 27(2), 406-415.
• Kılıç, S.F. (2022). Ölçek geliştirme sürecinde açımlayıcı faktör analizi [Exploratory factor analysis in the scale development process]. In M.Acar Güvendir & Y. Özer Özkan (Ed.), Tüm yönleriyle ölçek geliştirme süreci [Scale development process in all its aspects] (s.69-126). Pegem.
• King, F. (2014). Evaluating the impact of teacher professional development: An evidence-based framework. Professional Development in Education, 40(1), 89-111.
• Kirkpatrick, D., & Kirkpatrick, J. (2006). Evaluating training programs: The four levels. Berrett-Koehler Publishers.
• Kline, R.B. (2016). Yapısal eşitlik modellemesi ve uygulaması [Structural equation modeling and its application] (S. Şen, Çev. Ed.). Nobel.
• Koskimäki, M., Mikkonen, K., Kääriäinen, M., Lähteenmäki, M.L., Kaunonen, M., Salminen, L., & Koivula, M. (2021). Development and testing of the Educators' Professional Development scale (EduProDe) for the assessment of social and health care educators' continuing professional development. Nurse Education Today, 98, 104657.
• Kwakman, K. (1999). Leren van docenten tijdens de beroepsloopbaan [Teacher learning throughout the career] [Unpublished doctoral dissertation]. University of Nijmegen, the Netherlands.
• Lawshe, C.H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563-575.
• Little, J.W. (1993). Teachers' professional development in a climate of educational reform. Educational Evaluation and Policy Analysis, 15(2), 129-151.
• Liu, S., Hallinger, P., & Feng, D. (2016). Supporting the professional learning of teachers in China: Does principal leadership make a difference? Teaching and Teacher Education, 59, 79-91.
• MacCallum, R.C., Browne, M.W., & Sugawara, H.M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130.
• Mourão, L., Porto, J. B., & Puente-Palacios, K. (2014). Evidence of validity of the perception of professional development scale. Psico-USF, 19, 73-85.
• Otrar, M., & Argın, F.S. (2015). A scale development study to determine the attitude of students towards social media. Journal of Research in Education and teaching, 4(1), 391-403.
• Özdamar, K. (2016). Eğitim, sağlık ve davranış bilimlerinde ölçek ve test geliştirme yapısal eşitlik modellemesi [Structural equation modeling for scale and test development in education, health and behavioral sciences]. Nisan.
• Özer, N., & Beycioglu, K. (2010). The relationship between teacher professional development and burnout. Procedia-Social and Behavioral Sciences, 2(2), 4928-4932.
• Özgüven, İ.E. (2015). Psikolojik testler [Psychological tests]. Nobel.
• Robson. C. (2017). Bilimsel araştırma yöntemleri: Gerçek dünya araştırması [Real world research] (Ş. Çınkır & N. Demirkasımoğlu. Çev. Ed.). Anı Publishing.
• Saberi, L. & Sahragard, R. (2019). Designing and validating teachers' professional development scale: Iranian EFL contexts in focus. International Journal of Instruction, 12(1), 1609-1626.
• Sancho, L., Brown, M., Gardezi, S., O’Hara, J., & Rodríguez-Conde, M.J. (2024) Developing culturally responsive school leaders in Ireland and Spain. The evolving role of professional development. Irish Educational Studies, 1 22. https://doi.org/10.1080/03323315.2024.2334710
• Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23-74.
• Schumacker, R.E., & Lomax, R.G. (2010). A beginner’s guide to structural equation modeling. Routledge Taylor ve Francis Group.
• Seçer, İ. (2017). SPSS ve LISREL ile pratik veri analizi: Analiz ve raporlaştırma [Practical data analysis with SPSS and LISREL: Analysis and reporting]. Anı Publishing.
• Shabani, M.B., Alibakhshi, G., Bahremand, A., & Karimi, A.R. (2018). In-service professional development scale for EFL teachers: A validation study. The International Journal of Humanities, 25(3), 63-78.
• Soine, K.M., & Lumpe, A. (2014). Measuring characteristics of teacher professional development. Teacher Development, 18(3), 303-333.
• Sönmez, V., & Alacapınar, F.G. (2016). Sosyal bilimlerde ölçme aracı geliştirme [Developing measurement tools in social sciences]. Pegem.
• Spillane, J.P., Healey, K., & Mesler Parise, L. (2009). School leaders’ opportunities to learn: A descriptive analysis from a distributed perspective. Educational Review, 61(4), 407-432.
• Şen, S. (2020). Mplus ile yapısal eşitlik modellemesi uygulamaları [Structural equation modeling applications with Mplus]. Nobel.
• Tabachnick, B.G., & Fidell, L.S. (2007). Using multivariate statistics (5th ed.). Allyn and Bacon.
• Tabachnick, B.G., & Fidell, L.S. (2001). Using multivariate statistics. Allyn and Bacon.
• Tatlıdil, H. (2002). Uygulamalı çok değişkenli istatistiksel analiz [Applied multivariate statistical analysis]. Akademi Matbaası.
• Tavsancil, E. (2002). Tutumlarin olculmesi ve SPSS ile veri analizi [Measuring attitudes and data analysis with SPSS]. Nobel.
• Tezbaşaran, A. (2008). Likert tipi ölçek hazırlama kılavuzu [Likert type scale preparation guide]. Üçüncü Sürüm e Kitap. Retrieved from https://www.academia.edu/1288035/Likert_Tipi_Ölçek_Hazırlama_Kılavuzu
• Thorndike, R.M. & Thorndike-Christ, T. (2017). Psikolojide ve eğitimde ölçme ve değerlendirme [Measurement and evaluation in psychology and education] (M. Otrar, Çev. Ed.). Nobel.
• Torff, B., Sessions, D., & Byrnes, K. (2005). Assessment of teachers’ attitudes about professional development. Educational and Psychological Measurement, 65(5), 820-830.
• Ural, A., & Kılıç, İ. (2013). Bilimsel araştırma süreci ve SPSS ile veri analizi [Scientific research process and data analysis with SPSS]. Detay.
• Wayne, A.J., Yoon, K.S., Zhu, P., Cronen, S., & Garet, M.S. (2008). Experimenting with teacher professional development: Motives and methods. Educational Researcher, 37(8), 469-479.
• Yenen, E.T., & Kılınç, H.H. (2021). Öğretmenlerin mesleki gelişim öz yeterlikleri ölçeği geçerlik ve güvenirlik çalışması [Validity and reliability study of teachers' professional development self-efficacy scale]. Turkish Journal of Social Research/Turkiye Sosyal Arastirmalar Dergisi, 25(2).
• Zepeda, S.J. (2011). Professional development: What works. Eye on Education.
• Zhu, H. (2015, November). A study on professional development scale for master of full-time education (PDSM-FE). In 2015 International Conference on Social Science, Education Management and Sports Education (pp. 168-173). Atlantis Press.