The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift

Hikmet Şevgin; Mehmet Şata

doi:10.17275/per.23.77.10.5

EN

The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift

Abstract

This study aimed to investigate the change in teacher candidates' oral presentation skills over time through self, peer, and teacher assessments using the rater drift method. A longitudinal descriptive research model was used as a quantitative research approach to achieve this aim. The study group consisted of 47 teacher candidates receiving formation education at a state university in the Eastern Anatolia Region and an instructor teaching the course. An analytical rubric was used as a data collection tool to evaluate the candidates' oral presentation skills. The data collection process lasted six weeks in total. Since the performance evaluation process aimed to examine the change over time, the many-facet Rasch model was used. When the findings of the study were examined, it was determined that the rater behavior of teacher candidates had statistically significant differences at the group level over time. It was found that 26 out of 48 peer raters had rater drift in their evaluations. It was also found that the majority of rater drift over time was positive, meaning that evaluators became more generous over time. Another result obtained in the study was that teacher assessment did not show rater drift over time, with similar ratings for six weeks. The study’s findings were discussed with previous studies in the literature, and recommendations were made to researchers.

Keywords

Rater drift, presentation skills, many-facet rasch, reliability, validity

References

Alaz, A., & Yarar, S. (2009, May). Classroom teachers' preferences and reasons in the measurement and evaluation process. I. International Education Research Congress. Canakkale Onsekiz Mart University, Canakkale.
Alici, D. (2010). Other measurement tools and methods used in evaluating student performance. In Tekindal S. (Ed.), Measurement and evaluation in education (pp. 127-168). Pegem Akademi Publishing.
Ananiadou, K., & Claro, M. (2009). 21St century skills and competences for new millennium learners in OECD countries. OECD education working papers, 41, OECD Publishing.
Arik, R. S., & Kutlu, O. (2013). Scaling the competency of teachers' measurement and evaluation field based on judge decisions. Journal of educational sciences research, 3(2), 163-196. https://doi.org/10.12973/jesr.2013.3210a
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, policy & practice, 5(1), 7-74. https://doi.org/10.1080/0969595980050102
Board of Education (2005). Introduction booklet of primary school grades 1-5 curriculum. Ministry of National Education.
Borkan, B. (2017). Rater severity drift in peer assessment. Journal of Measurement and evaluation in education and psychology, 8(4), 469-489. https://doi.org/10.21031/epod.328119
Boud, D. (2013). Enhancing learning through self-assessment. Routledge. https://doi.org/10.4324/9781315041520
Case, H. (1997). An examination of variation in rater severity over time: A study in rater drift. Objective measurement: Theory into practice, 5, 1-38.
Cepni, S. (2010). Introduction to research and project work. Celepler Publishing.

Colvin, S., & Vos, E. K. (1997). Authentic assessment models for statistics education. The assessment challenge in statistics education, 27-36.
Congdon, P. J., & MeQueen, J. (2000). The stability of rater severity in large‐scale assessment programs. Journal of educational measurement, 37(2), 163-178. https://doi.org/10.1111/j.1745-3984.2000.tb01081.x
Dikli, S. (2003). Assessment at a distance: Traditional vs. alternative assessments. Turkish online journal of educational technology-TOJET, 2(3), 13-19.
Dishon, G., & Gilead, T. (2021). Adaptability and its discontents: 21st-century skills and the preparation for an unpredictable future. British journal of educational studies, 69(4), 393-413. https://doi.org/10.1080/00071005.2020.1829545
Dogan, C. D., & Uluman, M. (2017). A Comparison of rubrics and graded category rating scales with various methods regarding raters' reliability. Educational sciences: Theory and practice, 17(2), 631-651. https://doi.org/10.12738/estp.2017.2.0321
Donnon, T., McIlwrick, J., & Woloschuk, W. (2013). Investigating the reliability and validity of self and peer assessment to measure medical students’ professional competencies. Creative education, 4(6A), 23-28. https://doi.org/10.4236/ce.2013.46A005
Duban, N., & Kucukyilmaz, E. A. (2008). Classroom teacher candidates' views on the use of alternative assessment techniques in application schools. Elementary education online, 7(3), 769-784.
Dunn, K. E., & Mulvenon, S. W. (2009). A critical review of research on formative assessments: The limited scientific evidence of the impact of formative assessments in education. Practical assessment, research, and evaluation, 14(1), 1-11. https://doi.org/10.7275/jg4h-rb87
Engelhard Jr, G., & Myford, C. M. (2003). Monitoring faculty consultant performance in the advanced placement English literature and composition program with a many‐faceted Rasch model. ETS Research Report Series, 2003(1), i-60. https://doi.org/10.1002/j.2333-8504.2003.tb01893.x
Erman-Aslanoglu, A. & Sata, M. (2023). Examining the rater drift in the assessment of presentation skills in secondary school context. Journal of measurement and evaluation in education and psychology, 14(1), 62-75. https://doi.org/10.21031/epod.1213969
Erman-Aslanoglu, A. (2017). Evaluation of an individual within a group: Peer and self-assessment. Bogazici university journal of education, 34(2), 35-50.
Erman-Aslanoglu, A. (2022). Examining the effects of peer and self-assessment practices on writing skills. International journal of assessment tools in education, 9(Special Issue), 179-196. https://doi.org/10.21449/ijate.1127815
Falchikov, N. (1995). Peer feedback marking: Developing peer assessment. Innovations in Education and training International, 32(2), 175-187. https://doi.org/10.1080/1355800950320212
Farrokhi, F., Esfandiari, R., & Dalili, M. V. (2011). Applying the many-facet Rasch model to detect centrality in self-assessment, peer-assessment and teacher assessment. World applied sciences journal, 15(11), 70-77.
Farrokhi. F., Esfandiari. R., & Schaefer. E. (2012). A many-facet Rasch measurement of differential rater severity/leniency in three types of assessment. JALT journal, 34(1). 79-101. https://doi.org/10.37546/JALTJJ34.1-3
Gelbal, S., & Kelecioglu, H. (2007). Teacher competency perceptions and problems encountered in measurement and evaluation methods. Hacettepe university journal of education, (33), 135-145.
Gocer, A., Arslan, S., & Cayli, C. (2017). Process-oriented complementary assessment tools and methods for determining student development in Turkish education. Suleyman Demirel university journal of social sciences institute, (28), 263-292.
Gomleksiz, M. N., Yetkiner, A., & Yildirim, F. (2011). Teachers’ views on the use of alternative assessment and evaluation techniques in life studies class. Education sciences, 6(1), 823-840.
Guler, N. (2012). Measurement and assessment in education. Pegem Akademi Publishing. https://doi.org/10.14527/9786053641247
Hafner, J., & Hafner, P. (2003). Quantitative analysis of the rubric as an assessment tool: an empirical study of student peer‐group rating. Int. J. Sci. Educ., 25(12), 1509-1528. https://doi.org/10.1080/0950069022000038268
Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. Routledge. https://doi.org/10.4324/9780203850381
Hamayan, E. V. (1995). Approaches to alternative assessment. Annual review of applied linguistics, 15, 212-226. https://doi.org/10.1017/S0267190500002695
Hambleton. R. K., Swaminathan. H., & Rogers. H. J. (1991). Fundamentals of item response theory. SAGE Publications.
Harik, P., Clauser, B. E., Grabovsky, I., Nungester, R. J., Swanson, D., & Nandakumar, R. (2009). An examination of rater drift within a generalizability theory framework. Journal of educational measurement, 46(1), 43-58. https://doi.org/10.1111/j.1745-3984.2009.01068.x
Hoskens, M., & Wilson, M. (2001). Real‐time feedback on rater drift in constructed‐response items: An example from the golden sate examination. Journal of educational measurement, 38(2), 121-145. https://doi.org/10.1111/j.1745-3984.2001.tb01119.x
Hoyt, W. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it?. Psychological methods, 5(1), 64–86. https://doi.org/10.1037/1082-989X.5.1.64
Karakaya, I. (2015). Comparison of self, peer and instructor assessments in the portfolio assessment by using many facet Rasch model. Journal of education and human development, 4(2), 182-192. https://doi.org/10.15640/jehd.v4n2a22
Kassim. A.N.L. (2007, June). Exploring rater judging behaviour using the many-facet Rasch model. Paper presented in the second biennial international conference on teaching and learning of english in asia: Exploring new frontiers (TELiA2). Universiti Utara. Malaysia.
Kilic, D., & Gunes, P. (2016). Self, peer, and teacher assessment with grading rubrics. Mehmet Akif Ersoy university journal of education faculty, 1(39), 58-69. https://doi.org/10.21764/efd.93792
Kim. Y., Park. I., & Kang. M. (2012). Examining rater effects of the TGMD-2 on children with intellectual disability. Adapted physical activity quarterly, 29(4). 346-365. https://doi.org/10.1123/apaq.29.4.346
Kooken, J., Welsh, M. E., McCoach, D. B., Miller, F. G., Chafouleas, S. M., Riley-Tillman, T. C., & Fabiano, G. (2017). Test order in teacher-rated behavior assessments: Is counterbalancing necessary?. Psychological assessment, 29(1), 98-109. https://doi.org/10.1037/pas0000314
Kosterelioglu, İ., & Celen, Ü. (2016). Evaluation of the effectiveness of self-assessment method. Ilkogretim online, 15(2), 671-681. https://doi.org/10.17051/io.2016.44304
Koyuncu, M. S. & Sata, M. (2023). Using ACER ConQuest program to examine multidimensional and many-facet models. International journal of assessment tools in education, 10(2), 279-302. https://doi.org/10.21449/ijate.1238248
Kutlu, O., Dogan, C.D., & Karakaya, I., (2010). Determination of student achievement: Performance-based and portfolio-based authentic assessment and evaluation practices. Pegem Akademi Publishing.
Lamprianou, I. (2006). The stability of marker characteristics across tests of the same subject and across subjects. Journal of applied measurement, 7(2), 192-205.
Leckie, G., & Baird, J. A. (2011). Rater effects on essay scoring: A multilevel analysis of severity drift, central tendency, and rater experience. Journal of educational measurement, 48(4), 399-418. https://doi.org/10.1111/j.1745-3984.2011.00152.x
Linacre. J. M. (1996). Generalizability theory and many-facet Rasch measurement. Objective measurement: Theory into practice, 3, 85-98.
Linacre. J.M. (2017). A user's guide to FACETS: Rasch-model computer programs. MESA Press.
Linn, R. L. (2008). Measurement and assessment in teaching. Pearson Education
Maier, A., Adams, J., Burns, D., Kaul, M., Saunders, M., & Thompson, C. (2020). Using performance assessments to support student learning: how district ınitiatives can make a difference. performance assessment case study series. Learning policy institute. i-68. Palo Alto. https://doi.org/10.54300/213.365
McLaughlin, K., Ainslie, M., Coderre, S., Wright, B., & Violato, C. (2009). The effect of differential rater function over time (DRIFT) on objective structured clinical examination ratings. Medical education, 43(10), 989-992. https://doi.org/10.1111/j.1365-2923.2009.03438.x
McNamara, T. F., & Adams, R. J. (1991). Exploring rater characteristics with Rasch techniques. In Selected papers of the 13th Language Testing Research Colloquium (LTRC). Educational Testing Service, International Testing and Training Program Office.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American psychologist, 50(9), 741-749. https://doi.org/10.1037/0003-066X.50.9.741
Modarresi, G., Jalilzadeh, K., Coombe, C., & Nooshab, A. (2021). Validating a test to measure translation teachers' assessment literacy. Journal of Asia TEFL, 18(4), 1503-1511. https://doi.org/10.18823/asiatefl.2021.18.4.31.1503
Mulqueen, C., Baker, D., & Dismukes, R. K. (2000, April). Using multifacet Rasch analysis to examine the effectiveness of rater training. In 15th Annual Conference for the Society for Industrial and Organizational Psychology. https://doi.org/10.1037/e540522012-001
Myford, C. M., & Wolfe, E. W. (2009). Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale category use. Journal of educational measurement, 46(4), 371-389. https://doi.org/10.1111/j.1745-3984.2009.00088.x
Nalbantoglu Yilmaz, F. (2017). Analysis of the rater effects on the scoring of diagnostic trees prepared by teacher candidates with the many-facet Rasch model. Online submission, 8(18), 174-184. https://doi.org/10.15345/iojes.2016.02.020
National Research Council. (2001). Classroom assessment and the national science education standards. National Academies Press.
Noonan, B., & Duncan, C. R. (2005). Peer and self-assessment in high schools. Practical assessment, research, and evaluation, 10(1), 1-8. https://doi.org/10.7275/a166-vm41
Oren, F. S., Ormanci, U., & Evrekli, E. (2014). The alternative assessment-evaluation approaches preferred by pre-service teachers and their self-efficacy towards these approaches. Educational sciences: Theory & practice, 11(3), 1690-1698.
Orlova, N. (2019). Student peer performance evaluation: importance of implementation for group work enhancement. Science and education a new dimension: Pedagogy and psychology, 26-29. https://doi.org/10.31174/SEND-PP2019-207VII84-05
Ozpinar, I. (2021). Self, peer, group, and instructor assessment: A glimpse through the window of teacher competencies. Cumhuriyet international journal of education, 10(3), 949-973. https://doi.org/10.30703/cije.754885
Palm, T. (2008). Performance assessment and authentic assessment: A conceptual analysis of the literature. Practical assessment, research, and evaluation, 13(4), 1-11. https://doi.org/10.7275/0qpc-ws45
Park, Y. S. (2011). Rater drift in constructed response scoring via latent class signal detection theory and item response theory. Columbia University. https://doi.org/10.7916/D8445TGR
Petra, T. Z. H. T., & Ab Aziz, M. J. (2020, April). Investigating reliability and validity of student performance assessment in higher education using Rasch model. In Journal of Physics: Conference Series 1529(4), 042088. IOP Publishing. https://doi.org/10.1088/1742-6596/1529/4/042088
Quellmalz, E. (1980). Problems in stabilizing the judgment process (Vol. CSE Report No. 136). Center for the Study of Evaluation.
Raju. N. S. (1990). Determining the significance of estimated signed and unsigned areas between two item response functions. Applied psychological measurement, 14(2), 197-207. https://doi.org/10.1177/014662169001400208
Raymond, M. R., Harik, P., & Clauser, B. E. (2011). The impact of statistically adjusting for rater effects on conditional standard errors of performance ratings. Applied Psychological measurement, 35(3), 235-246. https://doi.org/10.1177/0146621610390675
Rennert-Ariev, P. (2005). A theoretical model for the authentic assessment of teaching. Practical assessment, research, and evaluation, 10(2), 1-12. https://doi.org/10.7275/a7h7-4111
Sad, S. N., & Goktas, O. (2013). Examination of traditional and contemporary measurement and evaluation approaches of academic staff. Ege education journal, 14(2), 79-105.
Sata, M. & Karakaya, I. (2022). Investigating the impact of rater training on rater errors in the process of assessing writing skill. International journal of assessment tools in education, 9(2), 492-514. https://doi.org/10.21449/ijate.877035
Sata, M. (2020a). Quantitative research approaches. In E. Oğuz (Ed.), Research methods in education (pp. 77-98). Egiten Kitap Publications.
Sata, M. (2020b, November). Evaluation of university students' oral presentation skills by their peers. 13th International Education Community Symposium. Online. Turkey.
Shepard, L. A. (2000). The role of assessment in a learning culture. Educational researcher, 29(7), 4-14. https://doi.org/10.3102/0013189X029007004
Swaminathan. H., & Rogers. H.J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational measurement, 27(4), 361-370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x
Szőkol, I., Szarka, K., & Hargaš, J. (2022). The functions of educational evaluation. R&Amp;E-SOURCE, (S24). https://doi.org/10.53349/resource.2022.iS24.a1112
Tunkler, V. (2019). Investigation of the contribution of peer assessment to pre-service teachers' professional knowledge and skills. Marmara university Atatürk education faculty journal of educational sciences, 50(50), 206-221. https://doi.org/10.15285/maruaebd.525171
Uto, M. (2022). A Bayesian many-facet Rasch model with Markov modeling for rater severity drift. Behavior research methods, 1-19. https://doi.org/10.3758/s13428-022-01997-z
Uzun, A., & Yurdabakan, I. (2011). An investigation of elementary school students' attitudes towards self-assessment. Mehmet Akif Ersoy university journal of education faculty, 11(22), 51-69.
Wayda, V., & Lund, J. (2005). Assessing dispositions: An unresolved challenge in teacher education. Journal of physical education, recreation & dance, 76(1), 34-41. https://doi.org/10.1080/07303084.2005.10607317
Wesolowski, B. C., Wind, S. A., & Engelhard Jr, G. (2017). Evaluating differential rater functioning over time in the context of solo music performance assessment. Bulletin of the council for research in music education, (212), 75-98. https://doi.org/10.5406/bulcouresmusedu.212.0075
Wigglesworth, G. (1994). Patterns of rater behaviour in the assessment of an oral interaction test. Australian review of applied linguistics, 17(2), 77-103. https://doi.org/10.1075/aral.17.2.04wig
Wolfe, E. W., Moulder, B. C., & Myford, C. M. (1999, April). Detecting differential rater functioning over time (DRIFT) using a Rasch multi-faceted rating scale model. Annual Meeting of the American Educational Research Association. Montreal, Quebec, Canada.
Wolfe, E. W., Myford, C. M., Engelhard Jr, G., & Manalo, J. R. (2007). Monitoring reader performance and DRIFT in the AP® English literature and composition examination using benchmark essays. Research Report No. 2007-2. College Board.
Yildiz, S. (2018). Developing a self-assessment scale for fractions. Mustafa Kemal university journal of faculty of education, 2(3), 30-44.
Yurdabakan, I. (2012). The effect of peer and collaborative assessment training on pre-service teachers’ self-assessment skills. Education and science, 37(163), 190-202.
Zhu, W., & Cole, E. L. (1996). Many-faceted Rasch calibration of a gross motor instrument. Research quarterly for exercise and sport, 67(1), 24-34. https://doi.org/10.1080/02701367.1996.10607922

Details

Primary Language

English

Subjects

Other Fields of Education

Journal Section

Research Article

Authors

Hikmet Şevgin ^*
0000-0002-9727-5865
Türkiye

Mehmet Şata
0000-0003-2683-4997
Türkiye

Early Pub Date

August 22, 2023

Publication Date

September 1, 2023

Submission Date

May 23, 2023

Acceptance Date

August 8, 2023

Published in Issue

Year 2023 Volume: 10 Number: 5

DOI

https://doi.org/10.17275/per.23.77.10.5

IZ

https://izlik.org/JA83NJ52ZH

APA

Şevgin, H., & Şata, M. (2023). The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift. Participatory Educational Research, 10(5), 98-118. https://doi.org/10.17275/per.23.77.10.5

AMA

1.Şevgin H, Şata M. The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift. PER. 2023;10(5):98-118. doi:10.17275/per.23.77.10.5

Chicago

Şevgin, Hikmet, and Mehmet Şata. 2023. “The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift”. Participatory Educational Research 10 (5): 98-118. https://doi.org/10.17275/per.23.77.10.5.

EndNote

Şevgin H, Şata M (September 1, 2023) The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift. Participatory Educational Research 10 5 98–118.

IEEE

[1]H. Şevgin and M. Şata, “The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift”, PER, vol. 10, no. 5, pp. 98–118, Sept. 2023, doi: 10.17275/per.23.77.10.5.

ISNAD

Şevgin, Hikmet - Şata, Mehmet. “The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift”. Participatory Educational Research 10/5 (September 1, 2023): 98-118. https://doi.org/10.17275/per.23.77.10.5.

JAMA

1.Şevgin H, Şata M. The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift. PER. 2023;10:98–118.

MLA

Şevgin, Hikmet, and Mehmet Şata. “The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift”. Participatory Educational Research, vol. 10, no. 5, Sept. 2023, pp. 98-118, doi:10.17275/per.23.77.10.5.

Vancouver

1.Hikmet Şevgin, Mehmet Şata. The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift. PER. 2023 Sep. 1;10(5):98-118. doi:10.17275/per.23.77.10.5

Cited By

Applying Nociplastic Pain Criteria in Chronic Musculoskeletal Conditions: A Vignette Study

Journal of Clinical Medicine

https://doi.org/10.3390/jcm14041179

27707

Creative Commons Atıf 4.0 International Lisance

download?token=eyJhdXRoX3JvbGVzIjpbXSwiZW5kcG9pbnQiOiJqb3VybmFsIiwib3JpZ2luYWxuYW1lIjoicGVyIHNjb3B1cyBpc3RhdHRpc3Rpay5wbmciLCJwYXRoIjoiOTg5Ni9mN2E1LzU3ZDcvNmEyMmIyZmFlMGJkYjYuMDUyNjIyMDMucG5nIiwiZXhwIjoxNzgwNjYyNTM5LCJub25jZSI6IjcxYzczMjhhOTdkOWYxM2ExZjE2NWExYjQyMzMyOWJhIn0.g4uvJoi6Oy8W0gRoUNW-d5FO9m4u30pEA0l1h8DwzSE