Aydınlatıcı Program Değerlendirme Modeline Dayalı Bir Program Değerlendirme Ölçeğinin Geliştirilmesi: Geçerlik ve Güvenirlik Analizi

Abstract

Program değerlendirme, uygulanmakta olan programların etkililiğine ilişkin veri toplama araçlarına dayalı olarak karar verme süreci olarak ifade edilebilir. Bu bağlamda hazırlanan her bir programın etkililiğinin sınanması amacıyla değerlendirilmesi gerekmektedir. Bu araştırma, Aydınlatıcı Değerlendirme Modeli çerçevesinde Öğrenme Güçlüğü Destek Eğitim Programı (ÖGDEP) için öğretmen görüşlerini belirlemek amacıyla geçerli ve güvenilir bir ölçme aracı geliştirmeyi hedeflemektedir. Çalışma grubunu, pilot uygulama, Açımlayıcı Faktör Analizi (AFA) ve Doğrulayıcı Faktör Analizi (DFA) aşamalarında yer alan toplam 763 öğretmen oluşturmaktadır. Araştırma sürecinde, ölçeğin kapsam ve yapı geçerliğini test etmek için bir pilot uygulama yapılmış ve uzman görüşlerine başvurulmuştur. AFA ve DFA sonuçları doğrultusunda, 36 maddeden ve dört faktörden oluşan bir ölçek yapısı oluşturulmuştur. Aydınlatıcı Program Değerlendirme Modeli Program Değerlendirme Ölçeği (APDMPDÖ)’nün güvenirlik düzeyi, Cronbach Alfa katsayısı, test-tekrar test yöntemi ve bileşik güvenirlik değerleri ile belirlenmiştir. Ölçek maddelerinin ayırt edicilik özellikleri, alt ve üst gruplar arasındaki t-testi karşılaştırmaları ile değerlendirilmiştir. Madde analizi sonucunda, ölçek formunu oluşturan maddelerin ayırt edici özelliğe sahip olduğu belirlenmiştir.

Keywords

• Program değerlendirme
• Öğretmen
• Aydınlatıcı Program Değerlendirme Modeli
• Öğrenme Güçlüğü Destek Eğitim Programı
• Ölçek geliştirme

Illuminating Program Evaluation Model Based Scale Development: Validity and Reliability Assessment

Abstract

Program evaluation is a systematic process aimed at making informed decisions regarding the effectiveness of implemented programs by utilizing data collection tools. In this regard, it is crucial to evaluate each program to measure its overall effectiveness. The aim of this study is to create a valid and reliable tool for evaluating teachers' perceptions of the Learning Disability Support Education Programme (LDSEP), based on the Illuminative Evaluation Model. The research was conducted during the 2023-2024 academic year, with A pilot application was conducted, and expert feedback was obtained to evaluate the content and construct validity of the scale. Drawing from the results of Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA), a scale comprising thirty-six items distributed across four dimensions was developed. The reliability of the Illuminating Program Evaluation Model Program Evaluation Scale (IPEMPES) was assessed through Cronbach's alpha, test-retest, and composite reliability indices. Additionally, the discriminatory power of the scale items was analyzed by conducting t-test comparisons between the lower and upper groups. The item analysis results indicated that each item in the scale exhibited discriminative power. The CFA supported the four-dimensional model consisting of thirty-six items for the Programme Evaluation Scale. The findings of this study led to the conclusion that the Illuminating Program Evaluation Model Program Evaluation Scale (IPEMPES) is a valid and reliable tool. Consequently, it can be effectively utilized to assess teachers' perspectives on the evaluation of the Learning Disability Support Education Programme (LDSEP), grounded in the Illuminative Program Evaluation Model.

Keywords

• Programme evaluation
• Teacher
• Illuminative Program Evaluation Model
• Learning Disability Support Education Programme
• Scale development

References

1. Aktan, O. (2020). Determination of Educational Needs of Teachers Regarding the Education of Inclusive Students with Learning Disability. International Journal of Contemporary Educational Research, 7(1), 149-164.
2. Aktan, O. (2023). Kapsayıcı Eğitim. S.F.Büyükalan (Ed.), Öğrenme öğretme kuram ve yaklaşımları içinde (4.Baskı, s. 221-265). Pegem Akademi.
3. Alderman, L. (2015). Illuminative evaluation as a method applied to Australian Government policy borrowing and implementation in higher education. Evaluation Journal of Australasia, 15(1), 4-14.
4. American Psychological Association [APA] (2013). DSM-5 Diagnostic and Statistical Manual of Mental Disorders. American Psychiatric Association. Washington.
5. Armstrong, F., Armstrong, D., & Barton, L. (2016). Inclusive education: Policy, contexts and comparative perspectives. Routledge.
6. Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: a primer. Frontiers in public health, 6 (149), 1-18.
7. Burden, B. (2017). Illuminative evaluation. In Frameworks for practice in educational psychology: A textbook for trainees and practitioners, edited by B. Kelly, L. M. Woolfson, and J. Boyle, 291– 308. 2nd ed. London: Jessica Kingsley
8. Büyüköztürk, Ş. (2017). Sosyal bilimler için veri analizi el kitabı: İstatistik, araştırma deseni, SPSS uygulamaları ve yorum (22. baskı). Ankara: Pegem.

9. Carpenter, S. (2018). Ten steps in scale development and reporting: A guide for researchers. Communication methods and measures, 12(1), 25-44. https://doi.org/10.1080/19312458.2017.1396583
10. Carter, E. W., Moss, C. K., Asmus, J., Fesperman, E., Cooney, M., Brock, M. E., ... & Vincent, L. B. (2015). Promoting inclusion, social connections, and learning through peer support arrangements. Teaching Exceptional Children, 48(1), 9-18.
11. Carvalho, J., & Chima, F. O. (2014). Applications of structural equation modeling in social sciences research. American International Journal Of Contemporary Research, 4 (1), 6-11.
12. Cattell, R. B. (1966). The scree test for the number of factors. Multivariate behavioral research, 1(2), 245-276.
13. Chan, F., Lee, G. K., Lee, E. J., Kubota, C. & Allen, C. A. (2007). Structural equation modeling in rehabilitation counseling research. Rehabilitation Counseling Bulletin, 51 (1), 53- 66. Doi: 10.1177/00343552070510010701
14. Clark, C., Dyson, A., & Millward, A. (2005). Theorising special education. In Theorising special education (pp. 162-179). Routledge.
15. De Boer, A. A., Pijl, S. J., & Minnaert, A. (2010). Attitudes of parents towards inclusive education: a review of the literature. European Journal of Special Needs Education, 25 (2), 165-181. Doi: 10.1080/08856251003658694.
16. DeVellis, R. F. (2017). Scale Development: Theory and Applications (4th ed.). Thousand Oaks, CA: Sage.
17. Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British journal of psychology, 105(3), 399-412.
18. Erkuş, A. (2012). Psikolojide ölçme ve ölçek geliştirme. Ankara: Pegem Akademi Yayınları.
19. Fabrigar, L. R., & Wegener, D. T. (2012). Exploratory factor analysis. Oxford University Press.
20. Fan, Y., Chen, J., Shirkey, G., John, R., Wu, S. R., Park, H., & Shao, C. (2016). Applications of structural equation modeling (SEM) in ecological studies: an updated review. Ecological Processes, 5, 1-12.
21. Field, A. (2018). Discovering statistics using IBM SPSS Statistics. The USA: Sage
22. Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2010). Program evaluation: Alternative Approaches and Practical Guidelines (4th Edition). Pearson
23. Fletcher, J. M., Lyon, G. R., Fuchs, L. S., & Barnes, M. A. (2018). Learning disabilities: From identification to intervention. Guilford Publications.
24. Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Sage.
25. Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2014). Multivariate data analysis (1st Edition). The UK, Harlow: Pearson Education Limited.
26. Hajjar, S. T. (2018). Statistical analysis: Internal-consistency reliability and construct validity. International Journal of Quantitative and Qualitative Research Methods, 6(1), 27-38.
27. Haynes, S. N., Richard, D., & Kubany, E. S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological assessment, 7(3), 238‐247.
28. Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational research methods, 7(2), 191-205.
29. Henseler, J., & Sarstedt, M. (2013). Goodness-of-fit indices for partial least squares path modeling. Computational statistics, 28, 565-580.
30. Heubeck, B., & Neill, J. T. (2000). Confirmatory factor analysis and reliability of the mentalhealth inventory for australian adolescents. Psychological Reports, 87, 431- 440. Doi: 10.2466/pr0.2000.87.2.431.
31. Howard, M. C. (2016). A review of exploratory factor analysis decisions and overview of current practices: What we are doing and how can we improve?. International journal of human-computer interaction, 32(1), 51-62.
32. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling: a multidisciplinary journal, 6(1), 1-55.
33. Iacobucci, D. (2010). Structural equations modeling: Fit indices, sample size, and advanced topics. Journal of consumer psychology, 20(1), 90-98.
34. Johnson, E. S., Humphrey, M., Mellard, D. F., Woods, K., & Swanson, H. L. (2010). Cognitive processing deficits and students with specific learning disabilities: A selective metaanalysis of the literature. Learning Disability Quarterly, 33 (1), 3–18. Doi: 10.1177/073194871003300101.
35. Kline, R. B. (2011). Principles and practice of structural equation modeling: issues and practical considerations. New York: The Guildford Press.
36. Krippendorff, K. (2018). Content analysis: An introduction to its methodology. Sage publications.
37. Kurth, J. A., Lyon, K. J., & Shogren, K. A. (2015). Supporting students with severe disabilities in inclusive schools: A descriptive account from schools implementing inclusive practices. Research and Practice for Persons with Severe Disabilities, 40(4), 261-274.
38. Kyriazos, T. A., & Stalikas, A. (2018). Applied psychometrics: The steps of scale development and standardization process. Psychology, 9 (11), 2531-2560.
39. Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x
40. McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum.
41. MEB (2008).Özel Öğrenme Güçlüğü Destek Eğitim Programı. Özel Öğretim Kurumları Genel Müdürlüğü. Ankara
42. MEB (2021). Özel Öğrenme Güçlüğü Olan Bireyler İçin Destek Eğitim Programı. Ankara
43. Melekoğlu, M. A. (2019). Özel öğrenme güçlüğüne giriş. M. A. Melekoğlu & O. Çakıroğlu (Ed.), Özel öğrenme güçlüğü olan çocuklar içinde (s. 1 – 33). AnkaraVize Akademik.
44. McDowell, M. (2018). Specific learning disability. Journal of paediatrics and child health, 54(10), 1077-1083.
45. Mokken, R. J. (2011). A theory and procedure of scale analysis: With applications in political research (Vol. 1). Walter de Gruyter.
46. Morrison, J. T. (2009). Evaluating factor analysis decisions for scale design in communication research. Communication Methods and Measures, 3(4), 195–215.
47. Morrison, J. (2016). DSM-5’i kolaylaştıran ‘Klinisyenler için tanı rehberi’. Muzaffer Şahin (Çev Ed.). Ankara: Nobel.
48. Norman, G. (2010). Likert scales, levels of measurement and the “laws” of statistics. Advances in health sciences education, 15, 625-632.
49. Oliva, P. F. (2009). Developing the curriculum. New York: Pearson Allyn and Bacon
50. Ornstein, A. C. & Hunkins, F. P. (2016). Eğitim programı temeller, ilkeler ve sorunlar. (Çev. A. Arı). Eğitim Kitapevi.
51. Özdamar, K. (2016). Ölçek ve test geliştirme yapısal eşitlik modellemesi. Eskişehir: Nisan.
52. Özüdoğru, F., & Adıgüzel, O. C. (2016). Aydınlatıcı program değerlendirme modeli. Anadolu Üniversitesi Sosyal Bilimler Dergisi, 16(Özel Sayı), 25-34.
53. Parlett, M. & Hamilton, D. (1977). Evaluation as Illumination: A New Approach to the Study of Innovatory Programs. Parlett, M. & Dearden, G. (Eds.), Introduction to Illuminative Evaluation: Studies in Higher Education içinde (ss. 9-29). California: Pacific Soundings Press.
54. Pett, M.A., Lackey, N. R. & Sullivan, J. J. (2003). Making Sense of Factor Analysis: The Use of Factor Analysis for Instrument Development in Health Care Research. Sage, Thousand Oaks. Doi:10.4135/9781412984898.
55. Raubenheimer, J. E. (2004). An item selection procedure to maximise scalereliability and validity. SA Journal of Industrial Psychology, 30 (4), 59-64.
56. Richards, J. C. (2001). Curriculum development in language teaching. Cambridge university press.
57. Ross, S. J. (2009). Program evaluation. The handbook of language teaching, 756-778.
58. Shi, D., DiStefano, C., Maydeu-Olivares, A., & Lee, T. (2022). Evaluating SEM model fit with small degrees of freedom. Multivariate behavioral research, 57(2-3), 179-207.
59. Sindelar, P. T., Brownell, M. T., & Billingsley, B. (2010). Special education teacher education research: Current status and future directions. Teacher Education and Special Education, 33(1), 8-24.
60. Slee, R. (2018). Defining the Scope of Inclusive Education. Thought Piece for Global Education Monitoring Report. Paris: UNESCO
61. Stake, R. E. (2000). Program Evaluation, Particularly Responsive Evaluation. D. L. Stufflebeam, G. F.
62. Stufflebeam, D. (2001). Evaluation models. New directions for evaluation, 2001(89), 7-98.
63. Swanson, H. L., Harris, K. R., & Graham, S. (Eds.). (2013). Handbook of learning disabilities. Guilford press.
64. Şencan, H. (2005). Sosyal ve davranışsal ölçümlerde güvenilirlik ve geçerlilik. Ankara: Seçkin.
65. Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics. The USA: Pearson Education.
66. Tanujaya, B., Prahmana, R. C. I., & Mumu, J. (2022). Likert scale in social sciences research: Problems and difficulties. FWU Journal of Social Sciences, 16(4), 89-101.
67. Tavşancıl, E. (2014). Tutumların ölçülmesi ve SPSS ile veri analizi, Ankara: Nobel.
68. Ullman, J. B. (2006). Structural equation modeling: rewiewing the basics and moving forward. Journal of personality assessment, 87 (1), 35- 50. Doi: 10.1207/s15327752jpa8701_03.
69. Vieira A. L. (2011). Interactive LISREL in practice, getting started with a SIMPLIS Approach. London: Springer. Doi:10.1007/978-3-642-18044-6.
70. Warner, R. M. (2008). Apllied statistics, from bivariate through Multivariate Tecniques. The USA: SAGE Publications, Inc.
71. Williams, B., Onsman, A., & Brown, T. (2010). Exploratory factor analysis: A five-step guide for novices. Australasian journal of paramedicine, 8, 1-13.
72. Wulan, R., & Sanjaya, W. (2022). Developing Positive School Climate for Inclusive Education. Journal of Education for Sustainability and Diversity, 1(1), 54-66.
73. Yang, Y., & Green, S. B. (2011). Coefficient alpha: A reliability coefficient for the 21st century?. Journal of psychoeducational assessment, 29(4), 377-392.
74. Yazıcıoğlu, T. (2020). Destek eğitim odalarında görev yapan öğretmenlerin gözüyle destek eğitim odaları. Ankara Üniversitesi Eğitim Bilimleri Fakültesi Özel Eğitim Dergisi, 21(2), 273-297.
75. Yıldırım A, & Şimşek H. (2013). Sosyal Bilimlerde Nitel Araştırma Yöntemleri (9. Baskı). Ankara: Seçkin Yayıncılık.