Intergrader consistency reliability, achievement-referential validity, and expert-referential severity of adult student online peer grading

Abstract

The study aimed to identify the reliability, validity, and severity of online peer grading of adult students, which remains uncertain from previous studies. There were 322 graduate students participating in the online peer grading activities for their presentations and discussions in the “Seminar I” and “Seminar II” courses. Both the peers and the teacher used a grading scale developed in the present study as a rubric to grade the students’ presentations. The peer grading and teacher grading scores can be inquired about and acquired from the peer grading system. Pearson’s correlation was used to examine the consistencies between peer and teacher graded marks (i.e., intergrader consistency reliability) and between peer graded and term project marks (achievement-referential validity). Intraclass Correlation Coefficient (ICC) was also used to measure the agreement between the two grading methods. A dependent-samples t-test was conducted to examine the differences between peer-graded and teacher-graded marks (expert-referential severity). The results revealed that online peer grading had sufficient intergrader consistency reliability and expert-referential validity, but insufficient expert-referential severity, when teacher was regarded as the standard-referential index. Online peer grading had sufficient achievement-referential validity when the term project score served as the achievement-referential index. Implications and recommendations are also presented in terms of academic theory and educational practice.

Keywords

• Peer assessment
• Per grading2
• Reliability
• Severity
• Validity

References

1. Adamu, N. (2023). What is the point of assessments? In L. Waller & S. Waller (Eds.), Higher Education – Reflections from the Field – Volume 4. IntechOpen. https://doi.org/10.5772/intechopen.110379
2. Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Prentice Hall. https://psycnet.apa.org/record/1985-98423-000
3. Brennan, R.L. (2001). Generalizability theory. Springer. https://link.springer.com/book/10.1007/978-1-4757-3456-0
4. Brookhart, S.M. (2013). How to create and use rubrics for formative assessment and grading. ASCD. https://books.google.com.tw/books/about/How_to_Create_and_Use_Rubrics_for_Format.html?id=VTbyCyXs8qwC&redir_esc=y
5. Chen, W. (2017). Knowledge convergence among pre-service mathematics teachers through online reciprocal peer feedback. Knowledge Management & E-Learning: An International Journal, 9(1), 1-18. https://doi.org/10.34105/j.kmel.2017.09.001
6. Cho, K., Schunn, C.D., & Wilson, R.W. (2006). Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives. Journal of Educational Psychology, 98(4), 891-901. https://doi.org/10.1037/0022-0663.98.4.891
7. Cirit, D.K. (2021). An analysis of self-, peer, and teacher assessment within the scope of classroom teaching activities. International Journal or Education, 9(2), 150 63. https://doi.org/10.34293/education.v9iS2-Sep.4380
8. DeCarlo, L.T., Kim, Y.K., & Johnson, M.S. (2011). A hierarchical rater model for constructed responses, with a signal detection rater model. Journal of Educational Measurement, 48(3), 333-356. https://doi.org/10.2307/23018150

9. De Grez, L., Valcke, M., & Roozen, I. (2012). How effective are self-and peer assessment of oral presentation skills compared with teachers' assessments? Active Learning in Higher Education, 13(2), 129-142. https://doi.org/10.1177/1469787412441284
10. Double, K.S., McGrane, J.A., & Hopfenbeck, T.N. (2019). The impact of peer assessment on academic performance: A meta-analysis of control group studies. Educational Psychology Review, 32, 481-509. https://doi.org/10.1007/s10648-019-09510-3
11. Earl, L. (2003). Assessment as learning: Using classroom to maximize student learning. In W.D. Hawley (Ed.), The keys to effective schools: Educational reform as continuous improvement. Corwin Press. https://doi.org/10.4135/9781483329512.N6
12. Eckes, T. (2017). Rater effects: Advances in item response modeling of human ratings–Part I. Psychological Test and Assessment Modeling, 59(4), 443-452. https://www.psychologie-aktuell.com/fileadmin/download/ptam/4-2017_20171218/03_Eckes.pdf
13. Esfandiari, R. (2015). Rater errors among peer assessors: Applying the Many-Facet Rasch measurement model. Iranian Journal of Applied Linguistics, 18(2), 77 107. https://doi.org/10.18869/acadpub.ijal.18.2.77
14. Esfandiari, R., & Myford, C.M. (2013). Severity differences among self-assessors, peer assessors, and teacher assessors rating EFL essays. Assessing Writing, 18, 111 131. https://doi.org/10.1016/j.asw.2012.12.002
15. Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A Meta-analysis comparing peer and teacher Marks. Review of Educational Research, 70, article 287. https://doi.org/10.2307/1170785
16. Farmasari, S., Wardana, L.A., & Suryaningsih, H. (2025). Predictive validity of peer assessment in micro-teaching: Correlation with teacher ratings among Indonesian preservice EFL teachers. Journal Evaluation in Education, 6(4), 1406-1414. https://doi.org/10.37251/jee.v6i4.2113
17. Freeman, S., & Parks, J.W. (2010). How accurate is peer grading? CBE—Life Sciences Education, 9(4), 482-488. https://doi.org/10.1187/cbe.10-03-0017
18. Goss, H. (2022). Student learning outcomes assessment in higher education and in academic libraries: A review of the literature. The Journal of Academic Librarianship, 48(2), article 102485. https://doi.org/10.1016/j.acalib.2021.102485
19. Hsu, P.-L. & & Huang, K.-H. (2015). Evaluating online peer assessment as an educational tool for promoting self-regulated learning. In L. Wang, S. Uesugi, I. H. Ting, K. Okuhara, & K. Wang (Eds.), Multidisciplinary social networks research. Communications in computer and information science, vol 540 (pp.161-173). Springer. https://doi.org/10.1007/978-3-662-48319-0_13
20. Huerta, L.A., & Molina, J.A.L. (2023). Peer assessment reliability in an organic chemistry activity: Do students overrate their peers? Journal of Chemical Education, 100, 3200 3208. https://doi.org/10.1021/acs.jchemed.2c00945
21. Jones, I., & Alcock, L. (2014). Peer assessment without assessment criteria. Studies in Higher Education, 39(10), 1774-1787. https://doi.org/10.1080/03075079.2013.821974
22. Karakaya, İ. (2015). Comparison of self, peer and instructor assessments in the portfolio assessment by using many facet RASCH model. Journal of Education and Human Development, 4(2), 182-192. https://doi.org/10.15640/jehd.v4n2a22
23. Khonki, Z.A., & Sadeghi, K. (2012). The effect of assessment type (self vs. peer vs. teacher) on Iranian University EFL students' course achievement. Language Testing in Asia, 2(4), 47-74. https://doi.org/10.1186/2229-0443-2-4-47
24. Koo, T.K., & Li, M.Y. (2016). A guideline of selecting and reporting Intraclass Correlation Coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155 163. https://doi.org/10.1016/j.jcm.2016.02.012
25. Li, H., Xiong, Y., Hunter, C.V., Guo, X., & Tywoniw, R. (2020). Does peer assessment promote student learning? A meta-analysis. Assessment & Evaluation in Higher Education, 45(2), 193-211. https://doi.org/10.1080/02602938.2019.1620679
26. Li, L. (2017). The role of anonymity in peer assessment. Assessment & Evaluation in Higher Education, 42, 645-656. https://doi.org/10.1080/02602938.2016.1174766
27. Liu, X., & Li, L. (2014). Assessment training effects on student assessment skills and task performance in a technology-facilitated peer assessment. Assessment & Evaluation in Higher Education, 39, 275-292. https://doi.org/10.1080/02602938.2013.823540
28. Longford, N. (1995). Models for uncertainty in educational testing. Springer-Verlag. https://doi.org/978-1-4613-8463-2
29. Loureiro, P., & Gomes, M.J. (2023). Online peer assessment for learning: Findings from higher education students. Education Sciences, 13(3), article 253. https://doi.org/10.3390/educsci13030253
30. Lu, J., & Law, N. (2012). Online peer assessment: Effects of cognitive and affective feedback. Instructional Science, 40, 257-275. https://doi.org/10.1007/s11251-011-9177-2
31. Luo, H., Robinson, A.C., & Park, J.Y. (2014). Peer grading in a MOOC: Reliability, validity, and perceived effects. Journal of Asynchronous Learning Networks, 18(2), 1 14. https://doi.org/10.24059/olj.v18i2.429
32. Luria, G. (2019). Climate as a group level phenomenon: Theoretical assumptions and methodological considerations. Journal of Organizational Behavior, 40(9 10), 1055 1066. https://doi.org/10.1002/job.2417
33. Matsuno, S. (2017). Adoptability of peer assessment in ESL classroom. Creative Education, 8(8), 1292-1301. https://doi.org/10.4236/ce.2017.88091
34. Mazulfah, Faridi, A., Rukmini, D., & Mujiyanto, Y. (2024). Is the collaborative peer assessment model effective for genre-based writing in university? International Journal of Evaluation and Research in Education, 12(4), 2327-2334. https://doi.org/10.11591/ijere.v12i4.25640
35. McGarr, O., & Clifford, A.M. (2013). Just enough to make you take it seriously: Exploring students’ attitudes towards peer assessment. Higher Education, 65(6), 677 693. https://doi.org/10.1007/s10734-012-9570-z
36. Morris, R., Perry, T., & Wardle, L. (2021). Formative assessment and feedback for learning in higher education: A systematic review. Review of Education, 9(3), article e3292. https://doi.org/10.1002/rev3.3292
37. Mumpuni, K.E., Priyayi, D.F., & Widoretno, S. (2022). How do students perform a peer assessment? International Journal of Instruction, 15(3), 751-766. https://doi.org/10.29333/iji.2022.15341a
38. National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association. https://www.aera.net/Publications/Books/Standards-for-Educational-Psychological-Testing-2014-Edition
39. Nicol, D.J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218. https://doi.org/10.1080/03075070600572090
40. Panadero, E., Andrade, H., & Brookhart, S.M. (2018). Fusing self-regulated learning and formative assessment: A roadmap of where we are, how we got here, and where we are going. Educational Psychologist, 53(3), 193-211. https://doi.org/10.1080/00461520.2018.1460463
41. Panadero, E., Brown, G.T.L., & Strijbos, J.W. (2016). The future of student self-assessment: A review of known unknowns and potential directions. Educational Psychology Review, 28(4), 803-830. https://doi.org/10.1007/s10648-015-9350-2
42. Paul, U., Mantravadi, A., Shah, J., Shah, S., Mylavarapu, S.V., Rashid, M.P., & Gehringer, E. (2025). Scaling success: A systematic review of peer grading strategies for accuracy, efficiency, and learning in contemporary education. arXiv. https://arxiv.org/abs/2508.11677
43. Power, J.R., & Tanner, D. (2023). Peer assessment, self-assessment, and resultant feedback: An examination of feasibility and reliability. European Journal of Engineering Education, 48(4), 615-628. https://doi.org/10.1080/03043797.2023.2185769
44. Salehi, M., & Gholampour, S. (2022). Self, peer, and teacher assessment of oral presentations: An investigation of reliability and validity. MEXTESOL Journal, 46(2), 1 10. https://www.mextesol.net/journal/index.php?page=journal&id_article=46349
45. Schellekens, L.H., Bok, H.G., de Jong, L.H., van der Schaaf, M.F., Kremer, W.D., & van der Vleuten, C.P. (2021). A scoping review on the notions of assessment as learning (AaL), assessment for learning (AfL), and Assessment of learning (AoL). Studies in Educational Evaluation, 71(1), article 101094. https://doi.org/10.1016/j.stueduc.2021.101094
46. Stiggins, R., & Chappuis, J. (2016). An introduction to student-involved assessment for learning (7th ed.). Pearson Education. https://www.pearson.com/en-us/subject-catalog/p/introduction-to-student-involved-assessment-for-learning-an/P200000000879/9780133550559
47. Sudirman, S., Sarjan, M., Rokhmat, J., Hamidi, H., & Fauzi, I. (2022). Multidimensional science education on performance assessment comprehensively with collaborative project learning based model: Philosophy perspective. Journal of Science and Science Education, 3(2), 96-102. https://doi.org/10.29303/jossed.v3i2.2278
48. Suen, H.K. (2014). Peer assessment for massive open online courses (MOOCs). The International Review of Research in Open and Distributed Learning, 15(3), 312 327. https://doi.org/10.19173/irrodl.v15i3.1680
49. Sung, Y.-T., Chang, K.-E., Chang, T.-H., & Yu, W.-C. (2010). How many heads are better than one? The reliability and validity of teenagers' self- and peer assessments. Journal of Adolescence, 33(1), 135-145. https://doi.org/10.1016/j.adolescence.2009.04.004
50. Sung, Y.T., Chang, K.E., & Liu, T.C. (2023). The effects of integrating peer assessment into project-based learning on students’ learning achievement and self-regulation. Studies in Educational Evaluation, 78, article 101248. https://doi.org/10.1016/j.stueduc.2023.101248
51. Syzdykova, Z., Koblandin, K., Mikhaylova, N., & Akinina, O. (2021). Assessment of e-portfolio in higher education. International Journal of Emerging Technologies in Learning, 16(2), 120-134. https://doi.org/10.3991/ijet.v16i02.18819
52. Tai, J., Ajjawi, R., Boud, D., Dawson, P., & Panadero, E. (2018). Developing evaluative judgement: Enabling students to make decisions about the quality of work. Higher Education, 76(3), 467-481. https://doi.org/10.1007/s10734-017-0220-3
53. Topping, K. (2021). Peer assessment: Channels of operation. Education Sciences, 11(3), article 91. https://doi.org/10.3390/educsci11030091
54. Tursynbayeva, N., Öç, U., & Karakaya, İ. (2024). The effect of rater training on rating behaviors in peer assessment among secondary school students. International Journal of Assessment Tools in Education, 11(3), 507-523. https://doi.org/10.21449/ijate.1438798
55. van Zundert, M., Sluijsmans, D., & van Merrienboer, J. (2010). Effective peer assessment processes: Research findings and future directions. Learning and Instruction, 20, 270-279. https://doi.org/10.1016/j.learninstruc.2009.08.004
56. Vygotsky, L.S. (1978). Mind in society: The development of higher psychological process. Harvard University Press. https://doi.org/10.2307/j.ctvjf9vz4
57. Wang, J., Gao, R., Guo, X., & Liu, J. (2020). Factors associated with students’ attitude change in online peer assessment–a mixed methods study in a graduate-level course. Assessment & Evaluation in Higher Education, 45(5), 714-727. https://doi.org/10.1080/02602938.2019.1693493
58. Yang, Y.F., & Tsai, C.C. (2010). Conceptions of and approaches to learning through online peer assessment. Learning and Instruction, 20(1), 72 83. https://doi.org/10.1016/j.learninstruc.2009.01.003
59. Yu, F.-Y., Liu, Y.-H., & Liu, K. (2023). Online peer assessment quality control: A proactive measure, validation study, and sensitivity analysis. Studies in Educational Evaluation, 78, article 10279. https://doi.org/10.1016/J.STUEDUC.2023.101279
60. Xiong, Y., Schunn, C.D., & Wu, Y. (2023). What predicts variation in reliability and validity of online peer assessment? A large-scale cross-context study. Journal of Computer Assisted Learning, 39(6), 2004-2024. https://doi.org/10.1111/jcal.12861
61. Zhang, Y., Chen, H., Pi, Z., & Yang, J. (2024). Interactive equality in peer assessment: The impacts on preservice teachers’ technology-enhanced learning design and feedback uptake. Teaching and Teacher Education, 138, article 104408. https://doi.org/10.1016/j.tate.2023.104408
62. Zhu, X., & Evans, C. (2022). Enhancing the development and understanding of assessment literacy in higher education. European Journal of Higher Education, 1 21. https://doi.org/10.1080/21568235.2022.2118149