Consistency Analysis Inter-Team and Inter-Year in Program Accreditation: TURAK Example from Türkiye

Year 2025, Volume: 15 Issue: Special Issue, 103 - 114, 30.06.2025

Celil Çakıcı

https://doi.org/10.53478/yuksekogretim.1542751

Abstract

Although it is relatively easy to ensure consistency within the team, it is very difficult to ensure consistency between teams and across years in accreditation assessments. In the study, an analysis was conducted based on the reports provided by evaluation teams to the Tourism Education Evaluation and Accreditation Board (TURAK) in Türkiye. A numeric rubric system—“1: Deficit, 2: Inadequate, 3: Acceptable, 4: Good, 5: Very good”—was used for fifty standards under nine dimensions. A written rule was established stating that the difference in scores should not exceed 2 for each item. The dimension average is calculated by summing the average scores of all items in the dimension and dividing by the number of items. A dataset was created using reports from the teams, covering the years 2021, 2022, and 2023. An analysis of the nine dimensions revealed that there were not any significant differences in team evaluations by year, university type, or evaluation type. This indicates that the teams made similar evaluations across years, universities, and evaluation types. In other words, this finding implies that consistency was achieved between teams across years, university type, and evaluation type.

Keywords

Program Accreditation, Consistency, Inter-Team Consistency, Inter-Year Consistency

Program Akreditasyonunda Takımlar ve Yıllar Arası Tutarlılık Analizi: Türkiye’den TURAK Örneği

Abstract

Akreditasyon değerlendirme takım içi tutarlılığı sağlamak nispeten kolay olsa da, takımlar arası ve yıllar arası tutarlılığı sağlamak oldukça zordur. Çalışmada, Türkiye’de Turizm Eğitimi Değerlendirme ve Akreditasyon Kurulu’na (TURAK) değerlendirme takımları tarafından verilen raporlar baz alınarak bir analiz gerçekleştirilmiştir. Dokuz boyut altında elli standart için “1: Eksik, 2: Yetersiz, 3: Kabul edilebilir, 4: İyi, 5: Çok iyi” şeklinde nümerik bir rubrik sistem kullanılmıştır. Her bir madde için puan farkının 2’yi geçmemesi gerektiği yazılı bir kural hâline getirilmiştir. Bir boyutun ortalaması, boyuttaki tüm maddelerin ortalama puanlarının toplanıp madde sayısına bölünmesiyle elde edilir. Takımlardan gelen raporlar dikkate alınarak 2021, 2022 ve 2023 yıllarına ait verileri kapsayan bir veri seti oluşturulmuştur. Dokuz boyut dikkate alınarak yapılan analizler; takımların yıllara, üniversite türüne ve değerlendirme türüne göre değerlendirmeleri arasında anlamlı bir fark olmadığını ortaya koymuştur. Bu bulgu, takımların yıllara, üniversitelere ve değerlendirme türüne göre benzer değerlendirmeler yaptığını ortaya koymaktadır. Bir başka deyişle, takımlar arasında yıllara, üniversite türüne ve değerlendirme türüne göre tutarlılık sağlandığı söylenebilir.

Keywords

Program Akreditasyonu, Tutarlılık, Takımlar Arası Tutarlılık, Yıllar Arası Tutarlılık

Thanks

Çalışma TURAK’ın ikincil verilerinden yararlanılarak hazırlanmıştır. Bu süreçte izin veren TURAK’a teşekkür ederim.

References

• Aktan, C. C., & Gencel, U. (2010). Accreditation in higher education. Journal of Organization and Management Sciences, 2(2), 137-146.
• Alpar, R. (2010). Uygulamalı istatistik ve geçerlik-güvenirlik. Detay Yayıncılık.
• Alves, N. D. C., von Wangenheim, C. G., Hauck, J. C. R., & Borgatto, A. F. (2020). A large-scale evaluation of a rubric for the automatic assessment of algorithms and programming concepts. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (pp. 556-562). https://doi.org/10.1145/3328778.3366840
• Arnanz, J., & Kaewnuch, K. (2019). Accreditation in tourism and hospitality undergraduate education in the ASEAN context: The case of Thailand from the Ted Qual perspective. The Journal of Social Sciences Research, 5(2), 275-283. https://doi.org/10.32861/jssr.52.275.283
• Bai, X., Xu, Y., & Ikem, F. (2013). Rubric and performance-based assessment. Issues in Information Systems, 14(2). https://doi.org/10.48009/2_iis_2013_1-11
• Bishop, W., Nespoli, O., & Parker, W. (2012). Rubrics for accreditation and outcomes assessment in engineering capstone projects. In Proceedings of the Canadian Engineering Education Association (CEEA).
• Çiçek, J. S., Mann, D., & Renaud, R. (2022). Supporting teaching practice, program improvement, and accreditation efforts in an engineering program. Canadian Biosystem Engineering Journal, 64, 9.1-9.11. https://doi.org/10.7451/CBE.2022.64.9.1
• Cura, F., & Ahmed Alani, T. (2018). Accreditation effect on quality of education at business schools. International Journal of Social Sciences & Educational Studies, 4(5), 71-82. https://doi.org/10.23918/ijsses.v4i5p71
• Dahal, S. (2022). Using rubrics in continuous assessment system: A narrative inquiry [Yayımlanmamış doktora tezi]. Kathmandu University.
• Fitriyani, N., Evendi, E., & Suwanto, S. (2024, Mart). The effect of using rubrics in improving the quality of assessment of mathematics learning. In Proceedings of International Seminar on Student Research in Education, Science, and Technology, 1(April 2024), 91-101.
• Gallardo, K. (2020). Competency-based assessment and the use of performance-based evaluation rubrics in higher education: Challenges towards the next decade. Problems of Education in the 21st Century, 78(1), 61-79. https://doi.org/10.33225/pec/20.78.61
• Greenfield, D., Pawsey, M., Naylor, J., & Braithwaite, J. (2013). Researching the reliability of accreditation survey teams: Lessons learned when things went wrong. Health Information Management Journal, 42(1), 4-10.
• Hash, R. (2019). Consistency in decision-making between survey teams and the decision-making body in a professional education program accrediting agency [Versiyon 1]. Med Ed Publish, 8, 113. https://doi.org/10.15694/mep.2019.000113.1
• Jaiswal, C. P. (2024). Modern method of curriculum. Assessment: Rubrics. Anusandhan Anveshika, XIV(Temmuz 2024), 21-30.
• Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130-144. https://doi.org/10.1016/j.edurev.2007.05.002
• Kılıçaslan, Ç. (2020). Higher education accreditation. PEYZAJ - Journal of Education, Science, Culture and Art Special Issue, 10-18.
• Kiesler, N., & Impagliazzo, J. (2022). Implementing diversity, equity, and inclusion in accreditation. In Proceedings of International Symposium on Accreditation of Engineering and Computing Education (ICACIT) (pp. 1-6). IEEE. https://doi.org/10.1109/ICACIT56139.2022.10041468
• Kiesler, N., & Impagliazzo, J. (2023). Evaluating diversity, equity, inclusion, and accessibility in accreditation. In Proceedings of International Symposium on Accreditation of Engineering and Computing Education (ICACIT) (pp. 1-5). IEEE. https://doi.org/10.1109/ICACIT59946.2023.10403684
• Kumar, P., Shukla, B., & Passey, D. (2020). Impact of accreditation on quality and excellence of higher education institutions. Investigación Operacional, 41(2), 151-167.
• Kurtay, M. Z., & Duran, A. (2018). Vakıf üniversitesinde akademisyen olmak: Rutinler ve bu rutinleri etkileyen faktörler. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 18(4), 2518-2540.
• Kwiek, M. (2011, March). The public/private dynamics in Polish higher education: Demand-absorbing private growth and its implications. In Proceedings of Higher Education Forum, 8 (March), 37-59.
• Özçiçek, Y., & Karaca, A. (2019). Quality and accreditation in higher education institutions: Evaluation of engineering education programs. Fırat University International Journal of Economics and Administrative Sciences, 3(1), 114-149.
• Prøitz, T. S., Stensaker, B., & Harvey, L. (2004). Accreditation, standards and diversity: An analysis of EQUIS accreditation reports. Assessment & Evaluation in Higher Education, 29(6), 735-750. https://doi.org/10.1080/0260293042000227263
• Ragupathi, K., & Lee, A. (2020). Beyond fairness and consistency in grading: The role of rubrics in higher education. In C. S. Sanger & N. W. Gleason (Eds.), Diversity and inclusion in global higher education: Lessons from across Asia (pp. 73-95). Palgrave Macmillan. https://doi.org/10.1007/978-981-15-1628-3
• Sayre-Stanhope, D. (2005). Consistency of review of accreditation documents by commissioners of the Accreditation Review Commission on Education for the Physician Assistant. The Journal of Physician Assistant Education, 16(2), 73-78.
• Semerci, C. (2017). Development of the perception of accreditation (PA) scale: Validity and reliability study. Bartın University Faculty of Education Journal, 6(3), 1093-1104.
• Shabani, E. A., & Panahi, J. (2020). Examining consistency among different rubrics for assessing writing. Language Testing in Asia, 10, 1-25. https://doi.org/10.1186/s40468-020-00111-4
• Shryock, K., & Reed, H. (2009). ABET accreditation-best practices for assessment. American Society for Engineering Education, In Proceedings of 2008 ASEE Gulf-Southwest Annual Conference. The University of New Mexico – Albuquerque.
• Sin, C., Tavares, O., & Amaral, A. (2017). The impact of programme accreditation on Portuguese higher education provision. Assessment & Evaluation in Higher Education, 42(6), 860-871. https://doi.org/10.1080/02602938.2016.1203860
• Tierney, R., & Simon, M. (2004). What’s still wrong with rubrics: Focusing on the consistency of performance criteria across scale levels. Practical Assessment, Research, and Evaluation, 9(1). https://doi.org/10.7275/jtvt-wg68
• TURAK. (2020). Consistency guideline. https://drive.google.com/file/d/1aZAde2PbMX2-fQOLUnJ3gWikgXq-v6Vy/view
• TURAK. (2023). Higher education tourism programs evaluation and accreditation application principles directive. https://docs.google.com/document/d/16asbTD6gHezNuXgn1gEXaiYiCRRgB9If/edit
• Wilkerson, J. R. (2020). Rubrics meeting quality assurance and improvement needs in the accreditation context. Quality Assurance in Education, 28(1), 19-32. https://doi.org/10.1108/QAE-04-2019-0045
• Yükseköğretim Kalite Kurulu. (2018, Kasım 23). Yükseköğretim Kalite Güvencesi ve Yükseköğretim Kalite Kurulu Yönetmeliği. T.C. Resmî Gazete, Sayı 30604. https://www.mevzuat.gov.tr/mevzuat?MevzuatNo=28996&MevzuatTur=7&MevzuatTertip=5