-

Abstract

The purpose of this study was to compare two different rating scales to be used in student evaluation of instructors in terms of psychometric properties, mainly halo and leniency effects, and user reactions. A behaviourally anchored rating scale (BARS) and a graphic rating scale (GRS) were developed and administered to a total sample of 270 college students (NBARS = 126; NGRS = 144) rating their real-life instructors. Although neither format had a clear psychometric advantage over the other, the GRS format was found to be slightly better than the BARS format in terms of resistance to halo. Furthermore, the BARS format received less favourable user reactions than did the GRS format. Considering the differential resources required for the development of a behaviour-based rating scale, the present results indicated the GRS format as being preferable to the BARS format. The implications of the findings are discussed along with suggestions for future research.

Keywords

• rating formats, student evaluations, user reactions, halo, leniency

Öğretim Elemanı Değerlendirmesinde Kullanılan Formatların Hale Etkisi, Cömertlik Etkisi ve Kullanıcı Tepkileri Açısından Karşılaştırılması

Abstract

Bu çalışmanın amacı öğrencilerin öğretim elemanı performansını değerlendirmede kullanılabilecek iki farklı değerlendirme formatını psikometrik özellikleri (hale ve cömertlik/aşırı olumluluk etkileri) ve kullanıcı tepkileri açısından karşılaştırmaktır. Bu amaçla geliştirilen davranış odaklı değerlendirme ölçeği (DODÖ) ve grafik değerlendirme ölçeği (GDÖ) toplam 270 üniversite öğrencisine (NBARS = 126; NGRS = 144) uygulanmış ve öğrencilerden gerçek hayattaki öğretim elemanlarının ders sırasındaki performansını değerlendirmeleri istenmiştir. Formatlardan hiçbiri psikometrik açıdan diğerinden belirgin bir şekilde üstün bulunmamakla birlikte GDÖ’nün hale etkisine dayanıklılık açısından DODÖ’den biraz daha iyi olduğu bulunmuştur. Ayrıca, GDÖ ile karşılaştırıldığında DODÖ kullanıcılardan daha az olumlu tepkiler almıştır. Davranış odaklı bir değerlendirme ölçeği geliştirmek için gereken farklı kaynaklar düşünüldüğünde GDÖ’nün DODÖ’ye tercih edilebileceği yönünde bulgular elde edilmiştir. Elde edilen bulguların sonuçları geleceğe yönelik önerilerle birlikte tartışılmaktadır.

Keywords

• değerlendirme formatları, öğrenci değerlendirmeleri, kullanıcı tepkileri, hale etkisi, cömertlik etkisi

References

1. Abrami, P. C., & d’Apollonia, S. (1990). The dimensionality of ratings and their use in personnel decisions. In M. Theall & J. Franklin (Eds.), Student ratings of instruction: Issues for improving practice (pp. 97-111). San Francisco: JosseyBass.
2. Bernardin, H. J. (2005). Behaviorally anchored rating scales. In Blackwell Encyclopedic Dictionary of Human Resource Management (pp. 22-23). Oxford, England: Blackwell Publishing.
3. Borman, W. C., & Dunnette, M. D. (1975). Behavior-based versus trait-oriented performance ratings: An empirical study. Journal of Applied Psychology, 60(5), 561-565.
4. Cooper, W. (1981). Ubiquitous halo. Psychological Bulletin, 90, 218-244.
5. D’Apollonia, S. & Abrami, P. C. (1997). Navigating student ratings of instruction. American Psychologist, 52(11), 1198120
6. Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51, 327-358.
7. Heckert, T. M., Latier, A., Ringwald, A., & Silvey, B. (2006). Relation of course, instructor, and student characteristics to dimensions of student ratings of teaching effectiveness. College Student Journal, 40(1), 195-203.
8. Jelley, R. B. & Goffin, R. D. (2001). Can performance-feedback accuracy be improved? Effects of rater priming and ratingscale format on rating accuracy. Journal of Applied Psychology, 86(1), 134-144.

9. Kingstrom, P. O., & Bass, A. R. (1981). A critical analysis of studies comparing behaviorally anchored rating scales (BARS) and other rating formats. Personnel Psychology, 34, 263-289.
10. Kline, T. J. B., & Sulsky, L. M. (2009). Measurement and assessment issues in performance appraisal. Canadian Psychology, 50(3), 161-171.
11. Krzystofiak, F., Cardy, R., & Newman, J. (1988). Implicit personality and performance appraisal: The influence of trait inferences on evaluations of behavior. Journal of Applied Psychology, 73(3), 515-521.
12. Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87(1), 72-107.
13. Landy, F. J., & Farr, J. L. (1983). The measurement of work performance: Methods, theory, and applications. New York: Academic Press.
14. Lowman, J. (1994). Professors as performers and motivators. College Teaching, 42(4), 137-141.
15. Madesen, C. K. & Napoles, J. (2006). A 30-year follow-up study of perceptions of students’ ratings of former instructors. UPDATE: Applications of Research in Music Education, 24(2), 45-53.
16. Marsh, H. W. & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52(11), 1187-1197.
17. MacDonald, H. A., & Sulsky, L. M. (2009). Rating formats and rater training redux: A context-specific approach for enhancing the effectiveness of performance management. Canadian Journal of Behavioural Science, 41(4), 2272
18. Milanowski, A. (2004). The relationship between teacher performance evaluation scores and student achievement: Evidence from Cincinnati. Peabody Journal of Education, 79(4), 33-53.
19. Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: SAGE Publications.
20. Roch, S. G., Sternburgh, A. M., Caputo, P. M. (2007). Absolute vs. relative performance rating formats: Implications for fairness and organizational justice. International Journal of Selection and Assessment, 15(3), 302-316.
21. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88(2), 413-428.
22. Schwab, D. P., Heneman III, H. G., & DeCotiis, T. A. (1975). Behaviorally anchored rating scales: A review of the literature. Personnel Psychology, 28, 549-562.
23. Smith, P. C., & Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47(2), 149-155.
24. Tziner, A. (1984). A fairer examination of rating scales when used for performance appraisal in a real organizational setting. Journal of Occupational Behaviour, 5, 103-112.
25. Tziner, A., Joanis, C., & Murphy, K. R. (2000). A comparison of three methods of performance appraisal with regard to goal properties, goal perceptions and ratee satisfaction. Group and Organization Management, 25, 175-190.
26. Tziner, A., & Kopelman, R. E. (2002). Is there a preferred performance rating format? A non-psychometric perspective. Applied Psychology: An International Review, 51(3), 479-503.
27. Tziner, A. & Kopelman, R. E. (1988). Effects of rating format on goal-setting dimensions: A field experiment. Journal of Applied Psychology, 73, 323-326.
28. Tziner, A., Kopelman, R. E., & Joanis, C. (1997). Investigation of raters’ and ratees’ reactions to three methods of performance appraisal: BOS, BARS, and GRS. Canadian Journal of Administrative Sciences, 14, 396-404.
29. Tziner, A., Kopelman, R. E., & Livneh, N. (1993). Effects of performance appraisal format on perceived goal characteristics, appraisal process satisfaction, and changes in rated job performance: A field experiment. Journal of Psychology, 127, 281-292.