Analysis of Peer and Self-Assessments Using the Many-facet Rasch Measurement Model and Student Opinions

Abstract

The aim of this study is to analyze the peer and self-assessments of higher education students' oral presentation skills with the many-facet Rasch measurement model and to determine students' opinions on peer and self-assessment. In the study, convergent parallel method, one of the mixed-method research approaches, was used. The study group consisted of 11 university students studying at a state university in the 2022-2023 academic year. The FACETS program was used to analyze the data. The three facets identified in the study were the assessee (11 students), the assessor (11 students) and the items (16 items). Therefore, 11 participants scored (peer and self-assessment) on a 16-item assessment form. In addition, students' opinions on peer and self-assessment were obtained through three open-ended interview questions prepared by the researcher. According to the results of the study, it was determined that there was a statistically significant difference between the students in terms of their oral presentation skills, between the assessors in terms of their strictness/generosity in scoring, and between the criteria (items) in terms of the level of difficulty in realization. In addition, the participant opinions obtained from each interview question were analyzed through themes and sub-themes formed according to the general thoughts on peer and self-assessment, experiences, and whether the participants considered themselves as a reliable rater or not. In terms of practice, it can be suggested to provide detailed and enlightening information to students before peer and/or self-assessment in the classroom environment, and to give quick feedback to those who have not done the assessment appropriately. In addition, the reasons for the biases identified in peer and self-assessments in the current study can be investigated in future studies.

Keywords

• Peer assessment
• Self-assessment
• Many-facet Rasch measurement model
• Oral presentation skills

References

1. Akın, Ö. ve Baştürk, R. (2012). The evaluation of the basic skills in violin training by many facet Rasch model. Pamukkale University Journal of Education, 31(1), 175-187. https://dergipark.org.tr/en/pub/pauefd/issue/11112/132860
2. Atılgan, H. (2005). Analysis of Special Ability sellection examination for music education department using many-facets Racsh measurement (İnönü University case). Eurasian Journal of Educational Measurement, 0(20), 62-73. https://web.s.ebscohost.com/ehost/pdfviewer/pdfviewer?vid=0&sid=1c78c0f3-93e0-43b5-9499-026878d2fcb1%40redis
3. Baştürk, R. (2008). Applying the many facet Rasch model to evaluate PowerPoint presentation performance in higher education. Assessment and Evaluation in Higher Education, 33(4), 431-444. https://doi.org/10.1080/02602930701562775
4. Baştürk, R. (2010). Bilimsel araştırma ödevlerinin çok yüzeyli Rasch ölçme modeli ile değerlendirilmesi. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 1(1), 51-57. https://dergipark.org.tr/en/pub/epod/issue/5808/77254
5. Boud, D., & Falchikov, N. (1989). Quantitative studies of student self-assessment in higher education: a critical analysis of findings. Higher Education, 18(5), 529-549. https://doi.org/10.1007/BF00138746
6. Brown, S. (2004). Assessment for learning. Learning and Teaching in Higher Education, 5(1), 81-89. https://eprints.glos.ac.uk/3607/
7. Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245-281. https://doi.org/10.3102/00346543065003245
8. Cheong, C. M., Luo, N., Zhu, X., Lu, Q., & Wei, W. (2023). Self-assessment complements peer assessment for undergraduate students in an academic writing task. Assessment & Evaluation in Higher Education, 48(1), 135-148. https://doi.org/10.1080/02602938.2022.2069225

9. Creswell, J. W. (2014). A concise introduction to mixed methods research. SAGE.
10. Crisp, V., & Ward, C. (2008). The development of a formative scenario-based computer assisted assessment tool in psychology for teachers: The PePCAA project. Computers & Education, 50(4), 1509-1526. https://doi.org/10.1016/j.compedu.2007.02.004
11. Eckes, T. (2005). Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly: An International Journal, 2(3), 197-221. https://doi.org/10.1207/s15434311laq0203_2
12. Elhan, A. H., & Atakurt, Y. (2005). Why is it necessary to use Rasch analysis when evaluating measures? Ankara Üniversitesi Tıp Fakültesi Mecmuası, 58(1), 47-50. https://doi.org/10.1501/Tipfak_0000000134
13. Evans, A. T., Mcnutt, R. A., Fletcher, S. W., & Fletcher, R. H. (1993). The characteristics of peer reviewers who produce good-quality reviews. Journal of General Internal Medicine, 8(8), 422-428. https://doi.org/10.1007/bf02599618
14. Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287-322. https://doi.org/10.3102/00346543070003287
15. Fete, M. G., Haight, R. C., Clapp, P., & McCollum, M. (2017). Peer evaluation instrument development, administration, and assessment in a team-based learning curriculum. American Journal of Pharmaceutical Education, 81(4), 1-10. https://doi.org/10.5688/ajpe81468
16. Gu, C. (2020). Student Peer Assessment. Review of Educational Theory, 3(2), 74-78. https://doi.org/10.30564/ret.v3i2.1762
17. Güler, N. (2014). Analysis of open-ended statistics questions with many facet Rasch model. Eurasian Journal of Educational Research, 55, 73-90. http://dx.doi.org/10.14689/ejer.2014.55.5
18. Gürlen, E. , Boztunç Öztürk, N. & Eminoğlu, E. (2019). Investigation of the reliability of teachers, self and peer assessments at primary school level with Generalizability Theory. Journal of Measurement and Evaluation in Education and Psychology, 10(4), 406-421. http://dx.doi.org/10.21031/epod.583891
19. Hanrahan, S. J., & Isaacs, G. (2001). Assessing self- and peer-assessment: The students' views. Higher Education Research & Development, 20(1), 53-70. https://doi.org/10.1080/07294360123776
20. Hoo, H. T., Deneen, C., & Boud, D. (2022). Developing student feedback literacy through self and peer assessment interventions. Assessment & Evaluation in Higher Education, 47(3), 444-457. https://doi.org/10.1080/02602938.2021.1925871
21. Köse, İ. A., Usta, H. G., & Yandı A. (2016). Evaluation of presentation skills by using many facets Rasch model. Bolu Abant İzzet Baysal University Journal of Faculty of Education, 16(4), 1853-1864. https://dergipark.org.tr/en/pub/aibuefd/issue/28550/304600
22. Linacre, J. M. (1993). Rasch-based generalizability theory. Rasch Measurement Transactions, 7(1), 283-284. https://www.rasch.org/rmt/rmt71h.htm
23. Linacre, J. M. (1994). Many-facet rasch model (2nd ed.). Mesa Press.
24. Linacre, J. M. (2023). A user’s guide to FACETS: Rasch-model computer programs (Program manual 3.85. 1). Chicago, IL. https://www.winsteps.com/a/Facets-Manual.pdf
25. Liu, N. F., and D. Carless. 2006. “Peer feedback: The learning element of peer assessment.” Teaching in Higher Education 11(3), 279-290. https://doi.org/10.1080/13562510600680582
26. Liu, E. Z.-F., Lin, S. S. J., Chiu, C.-H., & Yuan, S.-M. (2001). Web-based peer review: the learner as both adapter and reviewer. IEEE Transactions on Education, 44(3), 246-251. https://doi.org/10.1109/13.940995
27. Lu, J., & Law, N. (2011). Online peer assessment: effects of cognitive and affective feedback. Instructional Science, 40(2), 257-275. https://doi.org/10.1007/s11251-011-9177-2
28. Lundstrom, K., and W. Baker. 2009. “To give is better than to receive: The benefits of peer review to the reviewer’s own writing.” Journal of Second Language Writing 18(1), 30-43. https://doi.org/10.1016/j.jslw.2008.06.002
29. Mulqueen C., Baker D. & Dismukes R.K. (2000) Using multifacet Rasch analysis to examine the effectiveness of rater training. SIOP.
30. Mumpuni, K. E., Priyayi, D. F., & Widoretno, S. (2022). How do students perform a peer assessment? International Journal of Instruction, 15(3), 751-766. https://doi.org/10.29333/iji.2022.15341a
31. Nakamura, Y. (2000). Many-facet Rasch based analysis of communicative language testing results. Journal of Communication Students, 12, 3-13. https://eric.ed.gov/?id=ED449678
32. Nakamura, N. (2002). Teacher assessment and peer assessment in practice. Educational Studies, 44, 204-215. https://files.eric.ed.gov/fulltext/ED464483.pdf
33. Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218. https://doi.org/10.1080/03075070600572090
34. Orluwene, G. W., & Ekim, D. K. (2020). Promoting self-regulated learning through self-and peer-assessment techniques among secondary school students. International Journal of Arts and Commerce, 9(4), 1-16. https://ijac.org.uk/articles/9.4.1.1-16.pdf
35. Panadero, E., Alqassab, M., Fernández Ruiz, J., & Ocampo, J. C. (2023). A systematic review on peer assessment: intrapersonal and interpersonal factors. Assessment & Evaluation in Higher Education, 1-23. https://doi.org/10.1080/02602938.2023.2164884
36. Pantiwati, Y., & Husamah. (2017). Self and peer assessments in active learning model to increase metacognitive awareness and cognitive abilities. International Journal of Instruction, 10(4), 185-202. https://doi.org/10.12973/iji.2017.10411a
37. Prieto, G., & Nieto, E. (2014). Analysis of rater severity on written expression exam using many faceted Rasch measurement. Psicologica, 35(2), 385-397. https://www.redalyc.org/pdf/169/16931314011.pdf
38. Randall, J. & Engelhard, G. Jr. (2009). Examining teacher grades using Rasch measurement theory. Journal of Educational Measurement, 46(1), 1-18. https://doi.org/10.1111/j.1745-3984.2009.01066.x
39. Sande, J. C. G., & Godino-Llorente, J. I. (2014). Peer assessment and self-assessment: Effective learning tools in higher education. International Journal of Engineering Education, 30(3), 711-721. https://oa.upm.es/35804/
40. Semerci, Ç. (2011a). Doktora yeterlikler çerçevesinde öğretim üyesi, akran ve öz değerlendirmelerin Rasch ölçme modeliyle analizi. Journal of Measurement and Evaluation in Education and Psychology, 2(2), 164-171. https://dergipark.org.tr/en/pub/epod/issue/5804/77226
41. Semerci, Ç. (2011b). Analyzing microteaching applications with many-facet Rasch measurement model. Education and Science, 36 (161), 14-25. http://egitimvebilim.ted.org.tr/index.php/EB/article/view/145
42. Semerci, Ç., Semerci, N., & Duman, B. (2013). Analysis of seminar presentation performances of postgraduate students with many-facet Rasch model. The Journal of Sakarya Unıversity Education Faculty, 0(25), 7-22. https://dergipark.org.tr/tr/pub/sakaefd/issue/11221/133978
43. Stefani, L. A. J. (1994). Peer, self and tutor assessment: Relative reliabilities. Studies in Higher Education, 19(1), 69-75. https://doi.org/10.1080/03075079412331382153
44. Sudweeks, R. R., Reeve, S., & Bradshaw, W. S. (2004). A comparison of generalizability theory and many-facet Rasch measurement in an analysis of college sophomore writing. Assessing Writing, 9(3), 239-261. https://doi.org/10.1016/j.asw.2004.11.001
45. Topping, K. J., Smith, E. F., Swanson, I., & Elliot, A. (2000). Formative Peer Assessment of Academic Writing Between Postgraduate Students. Assessment & Evaluation in Higher Education, 25(2), 149-169. https://doi.org/10.1080/713611428
46. Tseng, S.-C., & Tsai, C.-C. (2007). On-line peer assessment and the role of the peer feedback: A study of high school computer course. Computers & Education, 49(4), 1161-1174. https://doi.org/10.1016/j.compedu.2006.01.007
47. Turner, J. (2003). Examining on art portfolio assessment using a many facet Rasch measurement model [Unpublished dissertation]. Boston College.
48. Uyanık, G. K., Güler, N., Teker, G. T., & Demir, S. (2019). The analysis of elementary science education course activities through many-facet Rasch model. Kastamonu Education Journal, 27(1), 139-150. https://doi.org/10.24106/kefdergi.2417
49. Wright, B.D., & Linacre, J.M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370-382. https://www.rasch.org/rmt/rmt83b.htm
50. Yan, Z., & Brown, G. T. (2017). A cyclical self-assessment process: Towards a model of how students engage in self-assessment. Assessment & Evaluation in Higher Education, 42(8), 1247-1262. https://doi.org/10.1080/02602938.2016.1260091
51. Yan, Z., Wang, X., Boud, D., & Lao, H. (2023). The effect of self-assessment on academic performance and the role of explicitness: a meta-analysis. Assessment & Evaluation in Higher Education, 48(1), 1-15. https://doi.org/10.1080/02602938.2021.2012644
52. Yüzüak, A. V., Yüzüak, B., & Kaptan, F. (2015). A many-facet Rasch measurement approach to analyze peer and teacher assessment for authentic assessment task. Journal of Measurement and Evaluation in Education and Psychology, 6(1), 1-11. https://doi.org/10.21031/epod.57425
53. Zhan, Y., Yan, Z., Wan, Z. H., Wang, X., Zeng, Y., Yang, M., & Yang, L. (2023). Effects of online peer assessment on higher‐order thinking: A meta‐analysis. British Journal of Educational Technology, 54, 817-835. https://doi.org/10.1111/bjet.13310