Examining The Rater Drift in The Assessment of Presentation Skills in Secondary School Context

Abstract

The alternative assessment, including peer assessment, helps students develop metacognition among the sub-categories of assessment types. Despite the advantage of alternative assessment, reliability and validity issues are the most significant problems in alternative assessment. This study investigated the rater drift, one of the rater effects, in peer assessment. The performance of 8 oral presentations based on group work in the Science and Technology course was scored by 7th-grade students (N=28) using the rubric researchers developed. The presentations lasted for four days, with two presentations each day. While examining the time-dependent drift in rater severity in peer assessment, the many-Facet Rasch Measurement model was used. Two indexes (interaction term and standardized differences) were calculated with many-facet Rasch measurement to determine the raters who made rater drift either individually or as a group. The analysis examined the variance of scores in the following days compared to the first day’s scores. Accordingly, the two methods used to determine rater drift gave similar results, and some raters at the individual level tended to be more severe or lenient over time. However, no significant rater drift at the group level showed that drifts had no specific models.

Keywords

• Alternative assessment
• many-facet Rasch measurement
• peer assessment
• validity
• rater drift

References

1. Andrade, H. G. (2005). Teaching with rubrics: The good, the bad, and the ugly. College Teaching, 53(1), 27-31. https://doi.org/10.3200/CTCH.53.1.27-31
2. Azarnoosh, M. (2013). Peer assessment in an EFL context: attitudes and friendship bias. Language Testing in Asia, 3(1), 1-10. https://doi.org/10.1186/2229-0443-3-11
3. Baird, J. A., Hayes, M., Johnson, R., Johnson, S., & Lamprianou, I. (2013). Marker effects and examination reliability. A Comparative exploration from the perspectives of generalisability theory, Rash model and multilevel modelling. Oxford University Centre for Educational Assessment. https://core.ac.uk/download/pdf/15171449.pdf
4. Batmaz, H., Türk, N., Kaya, A., & Yıldırım, M. (2022). Cyberbullying and cyber victimization: examining mediating roles of empathy and resilience. Current Psychology, 1-11. https://doi.org/10.1007/s12144-022-04134-3
5. Black, P., Harrison, C., & Lee, C. (2003). Assessment for learning: Putting it into practice. McGraw-Hill. http://www.mcgraw-hill.co.uk/html/0335212972.html
6. Börkan, B. (2017). Rater severity drift in peer assessment. Journal of Measurement and Evaluation in Education and Psychology, 8(4), 469-489. https://doi.org/10.21031/epod.328119
7. Bushell, G. (2006). Moderation of peer assessment in group projects. Assessment and Evaluation in Higher Education, 31(1), 91–108. https://doi.org/10.1080/02602930500262395
8. Congdon, P. J., & McQueen, J. (2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37(2), 163-178. https://psycnet.apa.org/doi/10.1111/j.1745-3984.2000.tb01081.x

9. Dochy, F. (2001). A new assessment era: different needs, new challenges. Learning and Instruction, 10(1), 11-20. https://doi.org/10.1016/S0959-4752(00)00022-0
10. Donnon, T., McIlwrick, J., & Woloschuk, W. (2013). Investigating the reliability and validity of self and peer assessment to measure medical students’ professional competencies. Creative Education, 4(6A), 23-28. http://dx.doi.org/10.4236/ce.2013.46A005
11. Erman Aslanoğlu, A., Karakaya, İ., & Şata, M. (2020). Evaluation of university students’ rating behaviors in self and peer rating process via many facet rasch model. Eurasian Journal of Educational Research, 20(89), 25-46. https://dergipark.org.tr/en/pub/ejer/issue/57497/815802
12. Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287–322. https://doi.org/10.2307/1170785
13. Farrokhi, F., Esfandiari, R. ve Dalili, M.V. (2011). Applying the Many-Facet Rasch Model to detect centrality in self-Assessment, peer-assessment and teacher assessment. World Applied Sciences Journal 15 (Innovation and Pedagogy for Lifelong Learning), 70-77. http://www.idosi.org/wasj/wasj15(IPLL)11/12.pdf
14. Farrokhi, F., Esfandiari, R., & Schaefer, E. (2012). A many-facet Rasch measurement of differential rater severity/leniency in three types of assessment. JALT Journal, 34(1), 79-101.
15. Hafner, J. C., & Hafner, P. M. (2003). Quantitative analysis of the rubric as an assessment tool: An empirical study of student peer-group rating. International Journal of Science Education, 25(12), 1509–1528. https://doi.org/10.1080/0950069022000038268
16. Harik, P., Clauser, B. E., Grabovsky, I., Nungester, R. J., Swanson, D., & Nandakumar, R. 2009). An examination of rater drift within a generalizability theory framework. Journal of Educational Measurement, 46(1), 43-58. https://doi.org/10.1111/j.1745-3984.2009.01068.x
17. Hauenstein, N. M. A. & McCusker, M. E. (2017). Rater training: Understanding effects of training content, practice ratings, and feedback. International Journal of Selection and Assessment, 25, 253-266. https://psycnet.apa.org/doi/10.1111/j.1745-3984.2009.01068.x
18. Humphris GM, & Kaney S. (2001). Examiner fatigue in communication skills objective structured clinical examinations. Medical Education, 35(5), 444-449. https://doi.org/10.1046/j.1365-2923.2001.00893.x
19. Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130-144. https://doi.org/10.1016/j.edurev.2007.05.002
20. Kaya, A., Türk, N., Batmaz, H., & Griffiths, M. D. (2023). Online gaming addiction and basic psychological needs among adolescents: the mediating roles of meaning in life and responsibility. International Journal of Mental Health and Addiction, 1-25. https://doi.org/10.1007/s11469-022-00994-9
21. Kassim, A.N.L. (2007, June 14-16). Exploring rater judging behaviour using the many-facet Rasch model [Conference Session]. The Second Biennial International Conference on Teaching and Learning of English in Asia: Exploring New Frontiers (TELiA2), Universiti Utara, Malaysia. http://repo.uum.edu.my/id/eprint/3212/
22. Kim, Y., Park, I., & Kang, M. (2012). Examining rater effects of the TGMD-2 on children with intellectual disability. Adapted Physical Activity Quarterly, 29(4), 346-365. https://doi.org/10.1123/apaq.29.4.346
23. Knoch, U., Read, J., & von Randow, J. (2007). Re-training writing raters online: How does it compare with face-to-face training?. Assessing Writing, 12(1), 26–43. https://doi.org/10.1016/j.asw.2007.04.001 Kubiszyn, T., & Borich, G. (2013). Educational testing and measurement: Classroom application and practice (10th ed.). John Wiley & Sons. https://l24.im/jV6yYCJ
24. Kutlu, Ö., Doğan, C. D., & Karaya, İ. (2014). Öğrenci başarısının belirlenmesi: Performansa ve portfolyoya dayalı durum belirleme [Determining student success: Assessment based on performance and portfolio]. Pegem. https://l24.im/k5cn
25. Leckie, G., & Baird, J. A. (2011). Rater effects on essay scoring: A multilevel analysis of severity drift, central tendency, and rater experience. Journal of Educational Measurement, 48(4), 399-418. https://doi.org/10.1111/j.1745-3984.2011.00152.x
26. Linacre, J.M. (2017). A user's guide to FACETS: Rasch-model computer programs. MESA Press.
27. Liu, H., & Brantmeier, C. (2019). “I know English”: Self-assessment of foreign language reading and writing abilities among young Chinese learners of English. System, 80, 60-72. https://doi.org/10.1016/j.system.2018.10.013
28. Lunz, M. E., & Stahl, J. A. (1990). Judge consistency and severity across grading periods. Evaluation & the Health Professions, 13(4), 425-444. https://psycnet.apa.org/doi/10.1177/016327879001300405
29. McLaughlin K, Ainslie M, Coderre S, Wright B & Violato C. (2009). The effect of differential rater function over time (DRIFT) on objective structured clinical examination ratings. Medical Education, 43(10), 989–992. https://doi.org/10.1111/j.1365-2923.2009.03438.x
30. Messick, S. (1994). Alternative modes of assessment, uniform standards of validity. ETS Research Report Series, 2, 1-22. https://doi.org/10.1002/j.2333-8504.1994.tb01634.x
31. Messick, S. (1996). Validity of performance assessments. In G. Phillips (Ed.), Technical issues in large-scale performance assessment (pp. 1–18). National Center for Education Statistics.
32. Moore, B. B. (2009). A consideration of rater effects and rater design via signal detection theory (Publication No. 3373803). [Doctoral dissertation, Columbia University]. ProQuest Dissertations & Theses Global.
33. Mulqueen C., Baker D. & Dismukes R.K. (2000, April). Using multifacet rasch analysis to examine the effectiveness of rater training [Conference Session]. 15th Annual Meeting of the Society for Industrial/Organizational Psychology, New Orleans, LA. https://www.air.org/sites/default/files/2021-06/multifacet_rasch_0.pdf
34. Myford, C. M. (1991, April 3-7). Judging acting ability: The transition from notice to expert [Conference Session]. Annual Meetin of the American Educational Research Association, Chicago IL. https://files.eric.ed.gov/fulltext/ED333032.pdf
35. Myford, C. M., & Wolfe, E. M. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Aplied Measurement, 4(4), 386-422. https://psycnet.apa.org/record/2003-09517-007
36. Myford, C. M., & Wolfe, E. M. (2009). Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale use. Journal of Educational Measurement, 46(4), 371-389. https://psycnet.apa.org/doi/10.1111/j.1745-3984.2009.00088.x
37. Oosterhof, A. (2003). Developing and using classroom assessment (3th ed.). Merrill/Prentice Hall. https://l24.im/OCKvkg2
38. Orsmond P, Merry S, Reiling K (2000) The use of student-derived marking criteria in peer and self-assessment. Assessment &Evaluation Higher Education, 25(1), 21–38. https://doi.org/10.1080/02602930050025006
39. Patri, M. (2002). The influence of peer feedback on self- and peer-assessment of oral skills. Language Testing, 19(2), 109–131. https://doi.org/10.1191/0265532202lt224oa
40. Pinot de Moira, A., Massey, C., Baird, J., & Morrissey, M. (2002). Marking consistency over time. Research in Education, 67(1), 79–87. https://doi.org/10.7227/RIE.67.8
41. Raju, N. S. (1990). Determining the significance of estimated signed and unsigned areas between two item response functions. Applied Psychological Measurement, 14(2), 197-207. https://psycnet.apa.org/doi/10.1177/014662169001400208
42. Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361-370. https://psycnet.apa.org/doi/10.1111/j.1745-3984.1990.tb00754.x
43. Şata. M. (2020a). Quantitative research approaches [Nicel araştırma yaklaşımları]. In Oğuz. E. (Ed.). Research methods in education [Eğitimde araştırma yöntemleri] (p. 77-98). Eğiten Kitap publishing.
44. Topping, K. (2003). Self and peer assessment in school and university: Reliability, validity and utility. In M. S. Segers, Dochy,R., & E. C. Cascallar (Eds.), In optimising new modes of assessment: In search of qualities and standards (pp. 55-87). Springer Dordrecht. https://doi.org/10.1007/0-306-48125-1
45. Topping, K. (2009). Peer assessment. Theory Into Practice, 48(1), 20-27. https://doi.org/10.1080/00405840802577569
46. Topping, K. (2017). Peer assessment: learning by judging and discussing the work of other learners. Interdisciplinary Education and Psychology, 1(1), 1-17. https://doi.org/10.31532/InterdiscipEducPsychol.1.1.007
47. Wolfe, E. W., Myford, C. M., Engelhard Jr. G., & Manalo, J. R. (2007). Monitoring reader performance and DRIFT in the AP® English literature and composition examination using benchmark essays. College Board. https://files.eric.ed.gov/fulltext/ED561038.pdf
48. Yurdabakan, İ., & Cihanoğlu, M. O. (2009). The effects of cooperative reading and composition technique with the applications of self and peer assessment on the levels of achivement, attitude and strategy use. Dokuz Eylul University The Journal of Graduate School of Social Sciences, 11(4), 105-123. https://l24.im/VbHg