A Program Evaluation Study for Measurement and Evaluation Course in Distance Education

Öz

This study aims to evaluate the online measurement and evaluation course in teacher training programs during the COVID-19 process. In the study, we sought answers to two primary research questions: What are the opinions of teachers and school administrators regarding their measurement and evaluation competencies? Does the online "measurement and evaluation" course have the qualities of an effective program in the "antecedents, transactions, and outcomes" dimension? We structured the research into two phases within a multistage evaluation design framework. The findings show that there were problems and positive aspects in all dimensions of the program. For example, adapting teacher training programs developed before COVID-19 to distance education processes was challenging. In distance education, some practices contradict the modern teaching and assessment approach. Such problems were reflected in teachers' acquisition of measurement and evaluation competencies. The achievement test we applied to the observed groups also confirmed these findings. For this reason, responsible organizations should not ignore the fact that we cannot renounce distance education. During program development, they should reconsider how the teachers will acquire measurement and evaluation competencies and how we will measure and evaluate in distance education.

Anahtar Kelimeler

• measurement
• evaluation
• distance education
• teacher training
• program

Uzaktan Eğitimde Ölçme ve Değerlendirme Dersine Yönelik Bir Program Değerlendirme Çalışması

Öz

Bu çalışmanın amacı, COVID-19 sürecinde öğretmen yetiştirme programlarında yer alan çevrimiçi eğitimde ölçme ve değerlendirme dersinin değerlendirilmesidir. Çalışmada iki temel araştırma sorusuna yanıt aranmıştır: Öğretmenlerin ve okul yöneticilerinin ölçme ve değerlendirme yeterliklerine ilişkin görüşleri nelerdir? Çevrimiçi “ölçme ve değerlendirme” dersi “girdiler, işlemler ve çıktılar” boyutlarında etkili bir programın sahip olması gereken nitelikleri taşıyor mu? Araştırma çok aşamalı bir değerlendirme tasarımı çerçevesinde iki aşamalı olarak yapılandırılmıştır. Araştırma bulguları, programın tüm boyutlarında sorunların ve olumlu yönlerin olduğunu göstermektedir. Örneğin, COVID-19 öncesi geliştirilen öğretmen yetiştirme programlarını uzaktan eğitim süreçlerine uyarlamak güçtür. Uzaktan eğitimde, bazı uygulamalar modern öğretim ve değerlendirme anlayışıyla çelişmektedir. Bu tür sorunlar öğretmenlerin ölçme ve değerlendirme yeterliklerini kazanmalarına yansımıştır. Gözlem yapılan gruplarda uygulanan başarı testi de bu bulguları doğrulamıştır. Bu nedenle sorumlu kuruluşlar uzaktan eğitimden vazgeçmenin artık mümkün olmadığını göz ardı etmemelidir. Ayrıca bu kuruluşların program geliştirme süreçlerinde uzaktan eğitimde ölçme ve değerlendirmenin nasıl yapılacağının yanı sıra öğretmenlerin ölçme ve değerlendirme yeterliklerini nasıl kazanacağını yeniden düşünmeleri gerekmektedir.

Anahtar Kelimeler

• ölçme
• değerlendirme
• uzaktan eğitim
• öğretmen yetiştirme
• program

Kaynakça

1. Ada, S., & Baysal, Z. N. (2013). Pedagojik-androgojik formasyon ve Türkiye'de öğretmen yetiştirme. Ankara: Pegem Academy Publication.
2. Akdemir, A. S. (2013). A history of teacher training programmes and their problems in Turkey. Electronic Turkish Studies, 8(12), 15-28.
3. Al-Shalout, M. I., Rasmi, M., & Hassan, M. A. (2021). Evaluation of E-exam during Covid-19. Psychology and Education Journal, 58(1), 4604-4612.
4. Alkharusi, H., Kazem, A. M., & Al-Musawai, A. (2011). Knowledge, skills, and attitudes of preservice and inservice teachers in educational measurement. Asia-Pacific Journal of Teacher Education, 39(2), 113-123.
5. Armstrong, R. A., Slade, S. V., & Eperjesi, F. (2000). An introduction to analysis of variance (ANOVA) with special reference to data from clinical experiments in optometry. Ophthalmic and Physiological Optics, 20(3), 235-241.
6. Atılgan, H. (2018). Değerlendirme ve not verme. In Atılgan, H. (Ed.) Eğitimde Ölçme ve Değerlendirme (11th Edition) (pp. 315-362). Anı Publication.
7. Aydın, Ö., & Dalkılıç, F. (2018). University student attendance management system. Journal of Science and Engineering, 20(60), 863-875.
8. Bae, S. H., Kim, J. H., & Lim, Y. W. (2009). A study on the effect of multimedia online learning contents on learner's performance. Journal of Korea Society of Digital Industry and Information Management, 5(1), 127-139.

9. Barthakur, A., Joksimovic, S., Kovanovic, V., Richey, M., & Pardo, A. (2022). Aligning objectives with assessment in online courses: Integrating learning analytics and measurement theory. Computers & Education, 104603.
10. Büyüköztürk, Ş. (2014). Sosyal bilimler için veri analizi el kitabıı. (3rd Edition). Ankara: Pegem Academy Publication.
11. Chan, C. K., & Luk, L. Y. (2021). A four-dimensional framework for teacher assessment literacy in holistic competencies. Assessment & Evaluation in Higher Education, 1-15.
12. Christ, T. J., & Desjardins, C. D. (2018). Curriculum-based measurement of reading: An evaluation of frequentist and Bayesian methods to model progress monitoring data. Journal of Psychoeducational Assessment, 36(1), 55-73.
13. Cohen, L., Manion, L., & Morrison, K. (2007). Research methods in education. Routledge. Conrad, D., & Openo, J. (2018). Assessment strategies for online learning: Engagement and authenticity. Athabasca University Press.
14. Council of Higher Education. (2018). Yeni öğretmen yetiştirme lisans programları. Retrieved July 15, 2020, from https://www.yok.gov.tr/kurumsal/idari-birimler/egitim-ogretim-dairesi/yeni-ogretmen-yetistirme-lisans-programlari
15. Council of Higher Education. (2007). Öğretmen yetiştirme ve eğitim fakülteleri (1982-2007). Retrieved November 26, 2020, from https://www.yok.gov.tr/Documents/Kurumsal/egitim_ogretim_dairesi/Ogretmen-Yetistirme/2.pdf
16. Council of Higher Education. (1998). Eğitim fakültesi öğretmen yetiştirme lisans programları. Retrieved November 26, 2020, from https://www.yok.gov.tr/Documents/Yayinlar/Yayinlarimiz/egitim-fakultesi-ogretmen-yetistirme-lisans-programlari-mart-1998.pdf.
17. Creswell, J. W. (2015). Educational research: Planning, conducting and evaluating quantitative and qualitative research (5th Edition). Pearson Prentice Hall.
18. Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd Edition). SAGE Publications, Inc.
19. Curaj, A., Deca, L., Egron-Polak, E., & Salmi, J. (2015). Higher education reforms in Romania: Between the Bologna process and national challenges. Springer Nature.
20. DeLuca, C., LaPointe-McEwan, D., & Luhanga, U. (2016). Teacher assessment literacy: A review of international standards and measures. Educational Assessment, Evaluation and Accountability, 28(3), 251-272.
21. Durak, G., Çankaya, S., & İzmirli, S. (2020). Examining the Turkish universities' distance education systems during the COVID-19 pandemic. Necatibey Faculty of Education Electronic Journal of Science and Mathematics Education, 14(1), 787-809.
22. Duman, S. N. (2019). Hizmet öncesi öğretmen eğitiminde öğretmenlik meslek bilgisi derslerinin değerlendirilmesi (Doctoral dissertation, Gazi University). Council of Higher Education, Thesis Center.
23. EPDAD. (2020). EPDAD teacher education standards. Retrieved May 4, 2020, from https://epdad.org.tr/icerik/epdads-teacher-education-standards
24. Ergül, A. Ö. (2019) Examination of teachers' assessment literacy levels in terms of some variables (Master's thesis, Hacettepe University). Council of Higher Education, Thesis Center.
25. Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines. (4th Edition). Pearson Education Inc.
26. Fuchs, L. S. (2017). Curriculum‐based measurement as the emerging alternative: Three decades later. Learning Disabilities Research & Practice, 32(1), 5-7.
27. Graves, M. N. (2010). Curriculum-based measurement attitudes and training practices in teacher education programs [Doctoral dissertation, Indiana University]. ProQuest Dissertations & Theses Global.
28. Grayson, T. E. (2018). Program evaluation, performance measures, and evaluability assessment in higher education. In Secolsky, C., & Denison, D. B. (Ed.) Handbook on measurement, assessment, and evaluation in higher education (2nd Edition) (pp. 616-639). Routledge.
29. Güler, N. (2018). Eğitimde ölçme ve değerlendirme. (12th Edition). Pegem Academy Publication. Güvendir, M. A., & Özkan, Y. Ö. (2016). Determining the perceptions of prospective teachers in relation to measurement and evaluation course through metaphors. Dumlupınar University Journal of Social Sciences, (47), 91-105.
30. Harmon, R. L., Sheehy, L. M., & Davis, D. M. (1998). The utility of external performance measurement tools in program evaluation. Rehabilitation Nursing, 23(1), 8-11.
31. Harris, C. (2017). A program evaluation of a technology based formative assessment for algebra readiness (Doctoral dissertation, The College of William and Mary in Virginia). ProQuest Dissertations & Theses Global.
32. Holmes, D. S. (1971). The teaching assessment blank: A form for the student assessment of college instructors. The Journal of Experimental Education, 39(3), 34-38.
33. Hurst, E. J. (2020). Web Conferencing and Collaboration Tools and Trends. Journal of Hospital Librarianship, 20(3), 266-279.
34. ISO, (2021). Teaching standards. Retrieved March 2, 2022, from https://www.iso.org/files/live/sites/isoorg/files/store/en/PUB100354.pdf
35. JCSEE. (2018). Checklist of the program evaluation standards statements. Retrieved November 26, 2020, from https://wmich.edu/evaluation/checklists
36. Jena, A. K., & Devi, J. (2020). Lockdown area of COVID-19: How does cartoon based e-contents effect on learning performance of Indian elementary school students with ADHD. Online Submission, 8(4), 189-201.
37. Karakuş, M. & Turhan Türkkan, B. (2017). Investigating the needs for measurement and evaluation course: A case study on English language teaching program. Journal of Education and Training Studies, 5(4), 227-239
38. Karaman, P. & Şahin, Ç. (2014). Öğretmen adaylarının ölçme değerlendirme okuryazarlıklarının belirlenmesi. Ahi Evran Ünv. Kırşehir Eğitim Fakültesi Dergisi (KEFAD), 15(2), 175-189.
39. Kaysi, F., & Aydemir, E., (2017). Assessment of interaction dimensions in distance education processes. The Journal of Social Science, 4(11), 778-790.
40. Kearns, L. R. (2012). Student assessment in online learning: Challenges and effective practices. Journal of Online Learning and Teaching, 8(3), 198-208.
41. Kınalıoğlu, İ. H., & Güven, Ş. (2011). Issues and solutions on measurement of student achievement in distance education. Proceedings of XIII. Academic Informatics Conference, Turkey, 637-644.
42. Koç, A. (2019). Fen bilgisi öğretmen adaylarının ölçme değerlendirme okuryazarlık düzeylerinin belirlenmesi (Master's thesis, Akdeniz University). Council of Higher Education, Thesis Center.
43. Sabancı, O. & Yazıcı, K. (2017). Öğretmen Adaylarının Ölçme ve Değerlendirmeye Yönelik Yeterlik Algılarının İncelenmesi. Trakya Üniversitesi Eğitim Fakültesi Dergisi, 7(1), 128-153.
44. Sevimel Şahin, A. (2019). İngiliz dili öğretmen adaylarının yabancı dilde ölçme değerlendirme okur yazarlığının incelenmesi (Doctoral dissertation, Anadolu University). Council of Higher Education, Thesis Center.
45. Koçoğlu, E. (2020). Analysis of course transfer systems used in distance education. The Eurasia Proceedings of Educational and Social Sciences, 18, 180-184.
46. Kozikoğlu, İ., & Kayan, M. F. (2018). Changes in education faculties from the viewpoints of academic staff: student profile, base score application and student selection. Kastamonu Education Journal, 26(6), 1863-1873.
47. Köse, E. (2012). Öğretimde ölçme ve değerlendirmenin planlanması. In Karip, E. (Ed.) Ölçme ve Değerlendirme (pp. 123-152). Pegem Academy Publication.
48. Kutluca Canbulat, A. N. (2014). An assessment related to instructors’ views about teachers eligibility and teacher candidates' selection process. Cumhuriyet International Journal of Education, 3(3), 1-11.
49. Landis, J, R., & Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.
50. McDowell, C. L. (1992). Standardized tests and program evaluation: Inappropriate measures in critical times. New Directions for Program Evaluation, 1992(53), 45-54.
51. Mehrens W. A., & Lehmann I. J. (1991). Measurement and evaluation in education and psychology (4th Edition). Wadsworth/Thomson Learning
52. Mertens, D. M., & Wilson, A. T. (2019). Program evaluation theory and practice. (2nd Edition). Guilford Publications.
53. Miles, H., Huberman, A. M., & Saldana, J. (2014). Qualitative data analysis: A methods sourcebook (3rd Edition). SAGE Publications, Inc.
54. Miller, M. D., Linn, R. L., & Gronlund, N. E. (2009). Measurement and assessment in teaching (10thEdition). Pearson Education, Inc.
55. Ministry of National Education. (2019). 2023 education vision. Retrieved July 15, 2020, from http://2023vizyonu.MNE.gov.tr/
56. Ministry of National Education. (2017). General competencies of the teaching profession. Ankara: General Directorate of Teacher Training and Development.
57. Ministry of National Education. (1980). Journal of papers numbered 2064. Ankara: Ministry of National Education General Directorate of Publications and Printed Educational Materials.
58. Ministry of National Education. (1975). Journal of papers numbered 1840. Ankara: Ministry of National Education General Directorate of Publications and Printed Educational Materials.
59. Mislevy, R. J. (2018). On measurement in educational assessment. In Secolsky, C., & Denison, D. B. (Ed.) Handbook on measurement, assessment, and evaluation in higher education (2nd Edition) (pp. 37-64). Routledge.
60. Morrow Jr, J. R., Jackson, A. W., Disch, J. G., & Mood, D.P. (2000). Measurement and Evaluation in Human Performance (2nd Edition). Human kinetics.
61. Noble, A. (2012). Examining the instructional utility of curriculum-based measurement in achievement evaluations (Doctoral dissertation, East Carolina University). ProQuest Dissertations & Theses Global.
62. OSYM (2021). 2018-YKS Yerleştirme sonuçlarına ilişkin sayısal bilgiler. Retrieved August 27, 2021, from https://www.osym.gov.tr/TR,15288/2018-yks-yerlestirme-sonuclarina-iliskin-sayisal-bilgiler.html
63. Özer, S., & Turan, E. Z. (2021). Opinions of prospective teachers about distance education due to COVID-19. Turkish Studies - Education, 16(2), 1049-1068.
64. Özkara, B. Ö. (2021). Determining the optimal duration of a single lecture in distance education using facial analysis of instructors. Turkish Online Journal of Educational Technology-TOJET, 20(2), 35-43.
65. Patton, M. Q. (2015). Qualitative research and evaluation methods. (4th Edition). SAGE Publications.
66. Pavlič, L., Beranič, T., Brezočnik, L., & Heričko, M. (2022). Towards a novel catalog of assessment patterns for distant education in the information technology domain. Computers & Education, 182, 104470.
67. Peters, O. (2013). Learning and teaching in distance education: analyses and interpretations from an international perspective. Routledge.
68. Pires Pereira, Í. S., Fernandes, E. L., Braga, A. C., & Flores, M. A. (2021). Initial teacher education after the Bologna process. Possibilities and challenges for a renewed scholarship of teaching and learning. European Journal of Teacher Education, 1-29.
69. Plake, B. S. (1993). Teacher assessment literacy: Teachers' competencies in the educational assessment of students. Mid-Western Educational Researcher, 6(1), 21-27.
70. QCT. (2021). Australian professional standards for teachers. Retrieved May 3, 2022, from https://www.qct.edu.au/standards-and-conduct/professional-standards
71. Raje, S., & Stitzel, S. (2020). Strategies for effective assessments while ensuring academic integrity in general chemistry courses during COVID-19. Journal of Chemical Education, 97(9), 3436-3440.
72. Salvucci, S., Walter, E., Conley, V., Fink, S., & Saba, M. (1997). Measurement error studies at the National Center for Education Statistics (NCES). Washington D. C.: U. S. Department of Education.
73. Safrit, M. J. (1990). Measurement and evaluation curricula in professional physical education programs—Current trends. Journal of Physical Education, Recreation & Dance, 61(3), 39-41.
74. Scriven, M. (1981). Evaluation thesaurus (3rd Edition). Edge Press.
75. Secolsky, C., & Denison, D. B. (Ed.) Handbook on measurement, assessment, and evaluation in higher education (2nd Edition). Routledge.
76. Shapiro, E. S., & Gebhardt, S. N. (2012). Comparing computer-adaptive and curriculum-based measurement methods of assessment. School Psychology Review, 41(3), 295-305.
77. Stake, R. E. (2014). Information science and responsive evaluation. E-Learning and Digital Media, 11(5), 443-450.
78. Stake, R. E. (2013). Responsive evaluation IV. In Alkin, M. C. (Ed.) Evaluation roots. A wider perspective of theorists' views and influences (pp. 189-197). SAGE Publications, Inc.
79. Stake, R. E. (2011). Program evaluation particularly responsive evaluation. Journal of Multi-Disciplinary Evaluation, 7(15), 180-201.
80. Stake, R. E. (2005). Qualitative case studies, In Norman K. D. ve Yvonna S. L. (Ed.) The sage handbook of qualitative research (3rd Edition) (pp. 443- 465). SAGE Publications, Inc.
81. Stake, R. E. (1967). The countenance of educational evaluation. Department for Exceptional Children, Gifted Children Section.
82. Stufflebeam, D. L. (1983). The CIPP model for program evaluation. In Madaus, G. F., Scriven, M., & Stufflebeam, D. L. (Ed.) Evaluation models: viewpoints on educational and human services evaluation (pp. 117-141). Kluwer Academic Publishers.
83. Stufflebeam, D. L., & Coryn, C. L. S. (2014). Evaluation theory, models, and applications. (2). San Francisco: Josey-Bass.
84. Tang, L. (2012). A formative program evaluation of treatment integrity practices, assessments and attitudes within a specialized school setting (Doctoral dissertation, University of Massachusetts Amherst). ProQuest Dissertations & Theses Global.
85. Tiedt, J. A., Owens, J. M., & Boysen, S. (2021). The effects of online course duration on graduate nurse educator student engagement in the community of inquiry. Nurse Education in Practice, 55, 1-8.
86. Tindal, G. (2013). Curriculum-based measurement: A brief history of nearly everything from the 1970s to the present. International Scholarly Research Notices, 2013, 1-29.
87. Tusing, M. E., & Breikjern, N. A. (2017). Using curriculum-based measurements for program evaluation: expanding roles for school psychologists. Journal of Applied School Psychology, 33(1), 43-66.
88. UK. Department for Education. (2021). Teachers' standards. Retrieved May 3, 2022, from https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1040274/Teachers__Standards_Dec_2021.pdf
89. US. Department for Education. (2021). Professional teaching standards. Retrieved May 3, 2020, from https://www.ed.gov/category/keyword/national-board-professional-teaching-standards
90. Ünver, G. (2016). Connection between the theory and practice in pre-service teacher education programs in Turkey. Journal of Higher Education and Science, (1), 61-70.
91. Valeyeva, N. S., Kupriyanov, R. V., & Valeyeva, E. R. (2015, September). Results and challenges of Russia's integration into Bologna Process. In 2015 International Conference on Interactive Collaborative Learning (ICL) (pp. 404-406). IEEE.
92. Veal, M. L. (1990). Measurement and evaluation curricula in professional physical education preparation programs - A view from the practitioner. Journal of Physical Education, Recreation & Dance, 61(3), 36-38.
93. Yaralı, D. (2017). Öğretmen adaylarının ölçme ve değerlendirmeye yönelik yeterlik algılarının incelenmesi (Kafkas Üniversitesi Örneği). Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 17(1), 487-504.
94. Yenen, E. T., & Durmaz, A. (2019). Investigation of teacher candidates' readiness levels for teaching profession. OPUS International Journal of Society Researches, 13(19), 922-940.
95. Zhang, Z. (1996). Teacher assessment competency: A Rasch model analysis. Paper presented at the Annual Meeting of the American Educational Research Association, April 8-12, New York. (ERIC Reproduction Service No. ED 400 322).