The Effect of Rater Training on Rater Behaviors in Rating Open-Ended Mathematics Items

Abstract

The purpose of this study was to investigate the effect of rater training on rater behaviors in the rating of open-ended items based on seventh-grade mathematics curriculum outcomes. The study was conducted using a non-equivalent control group pre-test/post-test design. The participant group consisted of 65 eighth-grade secondary school students and 20 pre-service mathematics teachers during the 2022–2023 academic year. Data were collected through a mathematics achievement test developed by the researcher, a holistic rating rubric, and a rater evaluation form. Raters in the experimental group received a four-week rater training program, and the data were analyzed using the many facet Rasch model. The findings revealed that prior to the training, raters in both the experimental and control groups exhibited behaviors of severity, leniency, the halo effect, and a tendency toward central tendency. Following the rater training, these behaviors were largely eliminated in the experimental group. In conclusion, rater training was found to reduce inconsistencies among raters, thereby contributing to the validity and reliability of the measurement outcomes. Based on these findings, it is recommended that raters receive training before rating open-ended items.

Keywords

• Open-ended items
• Rater behaviors
• Rater training
• Many-facet Rasch model
• Mathematics education

Matematik Dersindeki Açık Uçlu Soruların Puanlanmasında Puanlayıcı Eğitiminin Puanlayıcı Davranışlarına Etkisi

Abstract

Bu çalışmanın amacı, 7. sınıf matematik dersi kazanımlarına göre oluşturulan açık uçlu soruların puanlanmasında puanlayıcı eğitiminin puanlayıcı davranışları üzerindeki etkisini incelemektir. Araştırma, denk olmayan kontrol gruplu ön test–son test deseni ile yürütülmüş olup araştırmanın çalışma grubunu 2022–2023 eğitim-öğretim yılında sekizinci sınıfta öğrenimine devam eden 65 ortaokul öğrencisi ve 20 matematik öğretmeni adayı oluşturmaktadır. Veriler, araştırmacı tarafından geliştirilen matematik başarı testi, bütünsel dereceli puanlama anahtarı ve puanlayıcı değerlendirme formu aracılığıyla toplanmıştır. Deney grubundaki puanlayıcılara dört haftalık puanlayıcı eğitimi verilmiş ve veriler çok yüzeyli Rasch modeli kullanılarak analiz edilmiştir. Bulgular, eğitim öncesinde hem deney hem de kontrol grubundaki puanlayıcıların katılık, cömertlik, halo etkisi ve merkeze eğilim davranışları sergilediklerini göstermiştir. Puanlayıcı eğitimi sonrasında bu davranışlar deney grubunda büyük ölçüde ortadan kalkmıştır. Sonuç olarak, puanlayıcı eğitiminin puanlayıcılar arasındaki tutarsızlıkları azalttığı ve ölçme sonuçlarının geçerlik ve güvenirliğine katkı sağladığı bulunmuştur. Bu bulgulardan hareketle, açık uçlu soruların puanlanmasında puanlama işlemi öncesinde puanlayıcılara puanlayıcı eğitimi verilmesi önerilmektedir.

Keywords

• Açık uçlu maddeler
• Puanlayıcı davranışları
• Puanlayıcı eğitimi
• Çok yüzeyli Rasch modeli
• Matematik eğitimi

References

1. Ahmadi, A. (2019). A study of raters’ behavior in rating L2 speaking performance: Using rater discussion as a training tool. Issues in Language Teaching, 8(1), 195–224.
2. Anderson, L. W. & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Addison Wesley Longman.
3. Athey, T. R. & McIntyre, R. M. (1987). Effect of rater training on rater accuracy: Levels-of-processing theory and social facilitation theory perspectives. Journal of Applied Psychology, 72(4), 567–572.
4. Attali, Y. (2016). A comparison of newly trained and experienced raters on a standardized writing assessment. Language Testing, 33(1), 99–115.
5. Bernardin, H. J. & Buckley, M. R. (1981). Strategies in rater training. Academy of Management Review, 6(2), 205–212.
6. Bernardin, H. J. & Pence, E. C. (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology, 65(1), 60–66.
7. Bijani, H. (2018). Investigating the validity of an oral assessment rater training program: A mixed-methods study of raters’ perceptions and attitudes before and after training. Cogent Education, 5(1), Article 1460901. https://doi.org/10.1080/2331186X.2018.1460901
8. Bijani, H., Hashempour, B., & Said Bani Orabah, S. (2022). Development and validation of a training-embedded speaking assessment rating scale: A multifaceted Rasch analysis in speaking assessment. International Journal of Research in English Education, 7(3), 32–45.

9. Bijani, H. & Khabiri, M. (2017). Investigating the effect of training on raters’ bias toward test takers in oral proficiency assessment: A facets analysis. Journal of Asia TEFL, 14(4), 687–702.
10. Brijmohan, A. (2016). A many-facet Rasch measurement analysis to explore rater effects and rater training in medical school admissions. (Doktora Tezi). ProQuest Dissertations & Theses Global. https://www.proquest.com/dissertations-theses/many-facet-rasch-measurement-analysis-explore/docview/1855572428 sayfasından erişilmiştir.
11. Borman, W. C. (1975). Effects of instructions to avoid halo error on reliability and validity of performance evaluation ratings. Journal of Applied Psychology, 60(5), 556–560.
12. Brookhart, S. M. (2014). How to design questions and tasks to assess student thinking. ASCD.
13. Danielson, C. & Hansen, P. (2016). Performance tasks and rubrics for early elementary mathematics (2. b.). Routledge.
14. Doosti, M. & Ahmadi-Safa, M. (2021). Fairness in oral language assessment: Training raters and considering examinees’ expectations. International Journal of Language Testing, 11(2), 64–90.
15. Engelhard, G. Jr. (2013). Invariant measurement: Using Rasch models in the social, behavioral, and health sciences. Routledge. https://doi.org/10.4324/9780203073636
16. Feldman, J. M., Lazzara, E. H., Vanderbilt, A. A., & DiazGranados, D. (2012). Rater training to support high-stakes simulation-based assessments. Journal of Continuing Education in the Health Professions, 32(4), 279–286.
17. Gorman, C. A. & Rentsch, J. R. (2009). Evaluating frame-of-reference rater training effectiveness using performance schema accuracy. Journal of Applied Psychology, 94(5), 1336–1344.
18. Haladyna, T. M. (1997). Writing test items to evaluate higher-order thinking. Allyn & Bacon.
19. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Sage.
20. Hauenstein, N. M. A. & McCusker, M. E. (2017). Rater training: Understanding effects of training content, practice ratings, and feedback. International Journal of Selection and Assessment, 25(3), 253–266.
21. Hedge, J. W. & Kavanagh, M. J. (1988). Improving the accuracy of performance evaluations: Comparison of three methods of performance appraiser training. Journal of Applied Psychology, 73(1), 68–73.
22. Heinich, R. (1973). Technology and the management of instruction. Educational Communication and Technology Journal, 21(2), 224–224. https://doi.org/10.1007/BF02768950
23. Idris, M. & Zakaria, M. H. (2016). Gauging ESL learners’ CEFR ratings on oral proficiency in rater training. Man in India, 96(6), 1675–1682.
24. İlhan, M. & Gezer, M. (2023). Araştırmaların sınıflandırılması. B. Çetin, M. İlhan, & M. G. Şahin (Ed.), Eğitimde araştırma yöntemleri: Temel kavramlar, ilkeler ve süreçler (2. b.) içinde (s. 101–133). Pegem Akademi.
25. Kang, O., Rubin, D., & Kermad, A. (2019). The effect of training and rater differences on oral proficiency assessment. Language Testing, 36(4), 481–504.
26. Knoch, U., Fairbairn, J., Myford, C., & Huisman, A. (2018). Evaluating the relative effectiveness of online and face-to-face training for new writing raters. Papers in Language Testing and Assessment, 7(1), 61–86.
27. Kutlu, Ö., Doğan, C. D., & Karakaya, İ. (2017). Ölçme ve değerlendirme: Performansa ve portfolyoya dayalı durum belirleme (5. b.). Pegem Akademi.
28. Lane, S. & Stone, C. A. (2006). Performance assessment. R. L. Brennan (Ed.), Educational measurement (4. b.) içinde (s. 387–431). Praeger.
29. Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575.
30. Linacre, J. M. (1994). Sample size and item calibration stability. Rasch Measurement Transactions, 7(4), 328.
31. Linacre, J. M. (2023). FACETS computer program for many-facet Rasch measurement (Version 3.85.1) [Computer software]. https://www.winsteps.com/facets.htm sayfasından erişilmiştir.
32. May, G. L. (2008). The effect of rater training on reducing social style bias in peer evaluation. Business Communication Quarterly, 71(3), 297–313.
33. McNamara, T. F. (1996). Measuring second language performance. Longman.
34. Mohd-Noh, M. F. & Mohd-Matore, M. E. E. (2022). Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis. Frontiers in Psychology, 13, 941084. https://doi.org/10.3389/fpsyg.2022.941084
35. Moore, B. B. (2009). A consideration of rater effects and rater design via signal detection theory. (Doktora Tezi). ProQuest Dissertations & Theses Global. (Publication No. 3373803)
36. Myford, C. M. & Wolfe, E. W. (2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of Applied Measurement, 5(2), 189–227.
37. Pulakos, E. D. (1984). A comparison of rater training programs: Error training and accuracy training. Journal of Applied Psychology, 69(4), 581–588.
38. Rahman, A. A., Ahmad, J., Yasin, R. M., & Hanafi, N. M. (2017). Investigating central tendency in competency assessment of design electronic circuits: Analysis using many-facet Rasch measurement. International Journal of Information and Education Technology, 7(7), 525–528.
39. Rethinasamy, S. (2021). The effects of different rater training procedures on ESL essay raters’ rating accuracy. Pertanika Journal of Social Sciences & Humanities, 29, 14–27.
40. Romagnano, L. (2001). Implementing the assessment standards: The myth of objectivity in mathematics assessment. Mathematics Teacher, 94(1), 31–37.
41. Saal, F. E., Downey, R. G., & Lahey, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88(2), 413–428.
42. Saretsky, G. (1972). The OEO P.C. experiment and the John Henry effect. The Phi Delta Kappan, 53(9), 579–581.
43. Stamoulis, D. T. & Hauenstein, N. M. A. (1993). Rater training and rating accuracy: Training for dimensional accuracy versus training for ratee differentiation. Journal of Applied Psychology, 78(6), 994–1003.
44. Stalnaker, J. M. (1951). The essay type of examination. E. F. Lindquist (Ed.), Educational measurement (c. 1) içinde (s. 495–530). American Council on Education.
45. Sulsky, L. M. & Day, D. V. (1992). Frame-of-reference training and cognitive categorization: An empirical investigation of rater memory issues. Journal of Applied Psychology, 77(4), 501–510.
46. Şata, M. (2019). Performans değerlendirme sürecinde puanlayıcı eğitiminin puanlayıcı davranışları üzerindeki etkisinin incelenmesi. (Doktora Tezi). http://tez.yok.gov.tr sayfasından erişilmiştir.
47. Tajeddin, Z. & Alemi, M. (2014). Pragmatic rater training: Does it affect non-native L2 teachers’ rating accuracy and bias? International Journal of Language Testing, 4(1), 66–83. https://www.ijlt.ir/article_114394.html sayfasından erişilmiştir.
48. Veneziano, L. & Hooper, J. (1997). A method for quantifying content validity of health-related questionnaires. American Journal of Health Behavior, 21(1), 67–70.
49. Weigle, S. C. (1998). Using FACETS to model rater training effects. Language Testing, 15(2), 263–287.
50. Woehr, D. J. & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67(3), 189–205.
51. Zedeck, S. & Cascio, W. F. (1982). Performance appraisal decisions as a function of rater training and purpose of the appraisal. Journal of Applied Psychology, 67(6), 752–758.