Program Evaluation Experts’ Competencies: A Delphi Study

Abstract

Due to the diversity of steps to be followed in program evaluation and knowledge, skills, qualification and competency requirements, the evaluation should be a field of expertise similar to other disciplines. Thus, it is important to consider program evaluation as a field of expertise and to determine the required competencies. This study aimed to determine the competencies of program evaluation experts based on the views of curriculum and instruction experts. A Delphi study was carried out with 23 participants in the first round and each was selected via the purposive sampling method. In the first round of the Delphi study, the open-ended questions were sent to the participants and qualitative data were analyzed with content analysis. At the end of the first round, a 5-point Likert-type questionnaire with 82 items was developed. In the second round, participants were asked to answer questions and also the views of the participants on main and sub-themes and their comments on these concepts were analyzed. In the third round, participants were asked to reevaluate their responses. The study was finalized with an 82 itemed-questionnaire under four main themes: “theoretical professional competencies of program evaluation experts”, “practical professional competencies of program evaluation experts”, “professional values that program evaluation experts should have”, “professional skills that program evaluation experts should have”.

Keywords

• Program evaluation expert
• Program evaluation
• Curriculum and instruction
• Delphi Technique

References

1. Atik-Kara, D., Kürüm-Yapıcıoğlu, D., & Sever, D. (2020). Eğitim programları ve öğretim lisansüstü eğitim programlarının incelenmesi [Examination of curriculum and instruction graduate education programs]. Anadolu Üniversitesi Eğitim Fakültesi Dergisi, 4(2), 163-190.
2. Barnes, D. (1982). Practical curriculum study. London: Routhledge.
3. Bauer, M. W. (2000). Classical content analysis: A review. In M. W. Bauer & G. Gaskell (Eds.), In qualitative researching with text, ımage, and sound: A practical handbook (pp. 131-151). London: Sage Publications.
4. Büyüköztürk, Ş., Çakmak, K. E., Akgün, E. Ö., Karadeniz, Ş., & Demirel, F. (2010). Bilimsel araştırma yöntemleri [Scientific researh methods]. Ankara: Pegem Akademi.
5. Clayton, M. J. (1997). Delphi: A technique to harness expert opinion for critical decision‐making tasks in education. Educational Psychology, 17(4), 373-386.
6. Cooksy, L. J., & Mark, M. M. (2012). Influences on evaluation quality. American Journal of Evaluation, 33(1), 79-87.
7. Cousins, J. B., Goh, S. C., Elliott, C. J., & Bourgeois, I. (2014). Framing the capacity to do and use evaluation. In J. B. Cousins & I. Bourgeois (Eds.), Organizational capacity to do and use evaluation. New Directions for Evaluation, 141, 7-23.
8. Covert, R.W. (1992). Successful competencies in preparing professional evaluators. Paper presented at the annual meeting of the American Evaluation Association, Seattle, WA.

9. Creswell, J. W. (2008). Research design: Qualitative, quantitative and mixed methods approaches. Thousand Oaks, CA: Sage
10. Creswell, W. J., & Clark, L. V. (2007). Designing and conducting mixed method research. London Sage Publications.
11. Dajani, J.S., Sincoff, M.Z., & Talley, W.K. (1979). Stability and agreement criteria for the termination of Delphi studies. Technological Forecasting and Social Change, 13, 83-90.
12. Dalkey, N., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management Science, 9(3), 458-467.
13. Engle, M., Altschuld, J. W., & Kim, Y. (2006). 2002 Survey of evaluation preparation programs in universities: An update of the 1992 American Evaluation Association-sponsored study. American Journal of Evaluation, 27, 353-359.
14. Fletcher, A., & Childon, G. P. (2014). Using the Delphi method for qualitative, participatory action research in health leadership. International Journal of Qualitative Methods, 13, 1-18.
15. Habibi, A., Sarafrazi, A., & Izadyar, S. (2014). Delphi technique theoretical framework in qualitative research. The International Journal of Engineering and Science, 3(4), 8-13.
16. Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008-1015.
17. King, J. A. (2003). The challenge of studying evaluation theory. New Directions for Evaluation, 97, 57-66.
18. King, J. A., & Stevahn, L. (2015). Competencies for program evaluators in light of adaptive action: What? So what? Now what? In J. W. Altschuld & M. Engle (Eds.), Accreditation, certification, and credentialing: Relevant concerns for U.S. evaluators. New Directions for Evaluation, 145, 21–37.
19. King, J.A., Stevahn, L., Ghere, G., & Minnema, J. (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Evaluation, 22(2), 229-247.
20. LaVelle, J. M. & Donaldson, S. I. (2015). The state of preparing evaluators. In J.W. Altschuld & M. Engle (Eds.), Accreditation, certification, and credentialing: Relevant concerns for U.S. evaluators. New Directions for Evaluation, 145, 39–52.
21. McGuire, M., & Zorzi, R. (2005). Evaluator competencies and performance development. The Canadian Journal of Program Evaluation. 20(2), 73-99.
22. Melrose, M. (1998). Exploring paradigms of curriculum evaluation and concepts of quality. Quality in Higher Education, 4:1, 37-43.
23. Mertens, D.M. (1994). Training evaluators: Unique skills and knowledge. In J.W. Altschuld & M. Engle (Eds.), The preparation of Professional evaluators: Issues, perspectives, and programs. New Directions for Program Evaluation, 62, 17–27.
24. Miles, M. B., & Huberman, A. M. (1994). An expanded source book: Qualitative data analysis. London: Sage Publications.
25. Nworie, J. (2011). Using the Delphi technique in educational technology research. Tech Trends, 55(5), 24-30.
26. Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information & Management, 42(1), 15-29.
27. Oliva, P. F. (1988). Developing the curriculum. Glenview, III. : Scott, Foresman/Little, Brown College Division.
28. Ornstein, A.C., & Hunkins, F.P. (2009). Curriculum: Foundations, principles, and issues. USA: Pearson Education, Inc.
29. Özçelik, D. A. (1992). Eğitim programları ve öğretim [Curriculum and Instruction]. Ankara: ÖSYM Yayınları.
30. Powell, C. (2003). The Delphi technique: Myths and realities. Journal of Advanced Nursing, 41(4), 376-382.
31. Rowe, G., & Wright, G. (2001). Expert opinions in forecasting: The role of the Delphi Technique. In J. Armstrong (Ed.). Principles of Forecasting (pp. 125-144). Boston: Kluwer Academic.
32. Scheibe M, Skutsch M, Schofer J (2002). IV. C. Experiments in Delphi methodology. In: Linstone H, Turoff M (Eds.) The Delphi method: techniques and applications (pp 257-281).[Electronic version]. Newark, NJ: New Jersey Institute of Technology.
33. Scriven, M. (1996). The theory behind practical evaluation. Evaluation, 2, 393-404.
34. Scriven, M. (1996a). Types of evaluation and types of evaluator. American Journal of Evaluation, 17, 151-161.
35. Seuring, S., & Müller, M. (2008). Core issues in sustainable supply chain management–a Delphi study. Business Strategy and the Environment, 17(8), 455-466.
36. Skulmoski, G. J., Hartman, F. T., & Krahn, J. (2007). The Delphi method for graduate research. Journal of Information Technology Education, 6, 1-21.
37. Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005a). Establishing essential competenciesfor program evaluators. American Journal of Evaluation, 26(1), 43-59.
38. Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005). Evaluator competencies in university-based evaluation training programs. The Canadian Journal of ProgramEvaluation, 20(2), 101-123.
39. Şahin, A. E. (2009). Türkiye’ de ilköğretim okulu müdürlüğünün bir meslek olarak mevcut durumu: Bir Delphi çalışması [The state of elementary principalship as a profession in Turkey: A Delphi Study]. Pamukkale Üniversitesi Eğitim Fakültesi Dergisi, 26 (26), 125-136.
40. Şahin, A. E. (2001). Eğitim araştırmalarında Delphi tekniği ve kullanımı [Delphi technique and its uses in educational research]. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi 20: 215 - 220
41. Volkov, B. B. (2011). Beyond being an evaluator: The multiplicity of roles of the internal evaluator. In B. B. Volkov & M. E. Baron (Eds.), Internal evaluation in the 21st century. New Directions for Evaluation, 132, 25- 42.