TY  - JOUR
T1  - An analysis of scoring via analytic rubric and general impression in peer assessment
TT  - Akran değerlendirmede analitik rubrikle ve genel izlenimle yapılan puanlamaların incelenmesi
AU  - Boztunç Öztürk, Nagihan
AU  - Şahin, Melek Gülşah
AU  - İlhan, Mustafa
PY  - 2019
DA  - October
Y2  - 2019
DO  - 10.19128/turje.609073
JF  - Turkish Journal of Education
JO  - TURJE
PB  - Mehmet TEKEREK
WT  - DergiPark
SN  - 2147-2858
SP  - 258
EP  - 275
VL  - 8
IS  - 4
LA  - en
AB  - The aim of this researchwas to analyze and compare analytic rubric and general impression scoring inpeer assessment. A total of 66 university students participated in the studyand six of them were chosen as peer raters on a voluntary basis. In theresearch, students were supposed to prepare a sample study within the scope ofscientific research methods course and were also expected to present theirstudies in class. While the students were giving a presentation, their courseinstructor and peer raters conducted scoring, firstly by using the analyticrubric and subsequently by using general impressions. Collected data wereanalyzed using the Rasch model. Consequently, it was found that students weredistinguished from one another at a highly reliable rate using both scoringmethods. Additionally, it was discovered that the differences between students’ability levels were better revealed when analytic rubric was used. It wasascertained that there was a high level positive correlation between theability estimations obtained from the scores performed by the peers and theinstructor, regardless of the scoring method used. Finally, it was determinedthat ability estimations, corresponding peer raters’ analytic rubric andgeneral impression scoring, held a positive and highly strong relation.
KW  - Peer assessment
KW  - Analytic rubric scoring
KW  - General impression scoring
KW  - Many-facet Rasch model
N2  - Bu araştırmada, akrandeğerlendirmede analitik rubrikle ve genel izlenimle yapılan puanlamalarınkarşılaştırmalı olarak incelenmesi amaçlanmıştır. Araştırma 66 üniversiteöğrencisi üzerinde yürütülmüş ve bu öğrencilerden gönüllülük esasına dayalıolarak seçilen altısı, çalışmada akran değerlendirici olarak görev almıştır.Çalışmada, öğrencilerden bilimsel araştırma yöntemleri dersi kapsamında örnekbir çalışma hazırlamaları ve hazırladıkları çalışmayı sınıf ortamında sunmalarıistenmiştir. Öğrenciler sunum yaparken dersin sorumlu öğretim elemanı ile akrandeğerlendiriciler, çalışmaları önce analitik rubriğe göre ve ardından genelizlenimle puanlamıştır. Puanlamadan elde edilen veriler, Rasch modeline göreanaliz edilmiştir. Araştırmada, her iki puanlama yönteminde de bireylerinyüksek güvenirlikte birbirinden ayırt edildiği belirlenmiştir. Bununlabirlikte, analitik rubrik kullanıldığında bireylerin yetenek düzeyleriarasındaki farklılıkların daha hassas bir biçimde ortaya konulduğusaptanmıştır. Hem analitik rubrikle hem de genel izlenimle yapılandeğerlendirmede; akranlar ile öğretim elemanının verdiği puanlar üzerindenhesaplanan yetenek kestirimleri arasında, pozitif yönlü yüksek korelasyonlarbulunmuştur. Akranların analitik rubrikle ve genel izlenimle yaptıklarıpuanlamalara karşılık gelen yetenek kestirimlerinin pozitif yönlü güçlü birilişki içerisinde olduğu sonucuna varılmıştır.
CR  - Alharby, E. R. (2006). A comparison between two scoring methods, holistic vs. analytic using two measurement models, generalizability theory and the many facet Rasch measurement within the context of performance assessment. (Unpublished doctoral dissertation). Pennsylvania State University, Pennsylvania.
CR  - Alzaid, J. M. (2017). The effect of peer assessment on the evaluation process of students. International Education Studies, 10(6), 159–173. DOI: 10.5539/ies.v10n6p159
CR  - Amo, E., & Jareno, F. (2011). Self, peer and teacher assessment as active learning methods. Research Journal of International Studies, 18, 41–47.
CR  - Anita, H. (2011). The connection between analytic and holistic approaches to scoring in the writing component of the PROFEX EMP exam. Acta Medica Marisiensis, 57(3), 206–208.
CR  - Ashraf, H., & Mahdinezhad, M. (2015). The role of peer-assessment versus self-assessment in promoting autonomy in language use: A case of EFL learners. Iranian Journal of Language Testing, 5(2), 110–120.
CR  - Berry, R. (2008). Assessment for learning. Hong Kong: Hong Kong University Press.
CR  - Bostock, S. (2000). Student peer assessment. Retrieved from https://www.reading.ac.uk/web/files/engageinassessment/student_peer_assessment_-_stephen_bostock.pdf
CR  - Boud, D. (2013). Introduction: making the move to peer learning. Boud, D., Cohen, R., & Sampson, J. (Eds.), in Peer learning in higher education learning from & with each other (pp. 1–18). New York: Routledge.
CR  - Chi, E. (2001). Comparing holistic and analytic scoring for performance assessment with many–facet Rasch model. Journal of Applied Measurement, 2(4), 379–388.
CR  - Çetin, B., & Kelecioğlu, H.  (2004). The relation between scores predicted from structured features of essay and scores based on scoring key and overall impression in essay type examinations. Hacettepe University Journal of Education, 26, 19–26.
CR  - Falchikov, N. (2001). Learning together: Peer tutoring in higher education. London: Routledge Falmer.
CR  - Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287–322. DOI: 10.3102/00346543070003287
CR  - Frankland, S. (2007). Peer assessments among students in a problem based learning format. In S. Frankland (Eds.), Enhancing teaching and learning through assessment: Developing an appropriate model (pp. 144–155). Dordrecht: Springer.
CR  - Ghalib, T. K., & Al-Hattami, A. A. (2015). Holistic versus analytic evaluation of EFL writing: A case study. English Language Teaching, 8(7), 225–236. DOI: 10.5539/elt.v8n7p225
CR  - Gravells, A. (2014). The award in education and training. London: Learning Matters.
CR  - Gronlund, N. E. (1998). Assessment of student achievement. Boston: Allyn & Bacon.
CR  - Güler, N., İlhan, M., Güneyli, A., & Demir, S. (2017). An evaluation of the psychometric properties of three different forms of Daly and Miller’s writing apprehension test through Rasch analysis. Educational Sciences: Theory & Practice, 17(3), 721–744. DOI: 10.12738/estp.2017.3.0051
CR  - Hall, E. K., & Salmon, S. J. (2003). Chocolate chip cookies and rubrics helping students understand rubrics in inclusive settings. Teaching Exceptional Children, 35(4), 8–11. DOI: 10.1177/004005990303500401
CR  - Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: SAGE Publications, Inc.
CR  - Hanrahan, S. J., & Isaacs, G. (2001). Assessing self– and peer–assessment: The students' views. Higher Education Research & Development, 20(1), 53–70. DOI: 10.1080/07294360123776
CR  - Hinkle, D. E., Wiersma, W., & Jurs, S. G. (1979). Applied statistics for the behavioral sciences. Chicago: Rand McNally College.
CR  - Hunter, D. M., Jones, R. M., Radhawa, B. S. (1996). The use of holistic versus analytic scoring for large–scale assessment of writing. The Canadian Journal of Program Evaluation, 11(2), 61–85.
CR  - Jönsson, A., & Balan, A. (2018). Analytic or holistic: A study of agreement between different grading models. Practical Assessment, Research & Evaluation, 23(12), 1–11.
CR  - Kan, A. (2007). An alternative method in the new educational program from the point of performance-based assessment: Rubric scoring scales. Educational Sciences: Theory & Practice, 7(1), 144–152.
CR  - Karaca, E. (2009). An evaluation of teacher trainees’ opinions of the peer assessment in terms of some variables. World Applied Sciences Journal, 6(1), 123–128.
CR  - Knoch, U. (2009). Diagnostic assessment of writing: A comparison of two rating scales. Language Testing, 26(20), 275–304. DOI: 10.1177/0265532208101008
CR  - Lee, M., Peterson, J.  J., & Dixon, A. (2010). Rasch calibration of physical activity self-efficacy and social support scale for persons with intellectual disabilities. Research in Developmental Disabilities, 31(4), 903−913. DOI: 10.1016/j.ridd.2010.02.010
CR  - Lester, F. K., Lambdin, D. V., & Preston, R. V. (1997). A new vision of the nature and purposes of the assessment in the mathematics classroom. Phye, G. D. (Eds.) in Handbook of Classroom Assessment: Learning, Adjustment, and Achievement (pp. 287−320). San Diego, CA, US: Academic Press.
CR  - Linacre, J. M. (2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3(1), 85−106.
CR  - Linacre, J. M. (2018). A user's guide to FACETS Rasch-model computer programs. Retrieved from https://www.winsteps.com/a/Facets-Manual.pdf
CR  - Liu, N. F., & Carless, D. (2006). Peer feedback: the learning element of peer assessment. Teaching in Higher Education, 11(3), 279−290. DOI: 10.19080/13562510600680582
CR  - Macpherson, K. (1999). The development of critical thinking skills in undergraduate supervisory management units: Efficacy of student peer assessment. Assessment & Evaluation in Higher Education, 24(3), 273−284. DOI: 10.1080/0260293990240302
CR  - Mann, B. L. (2006). Testing the validity of the post and vote model of web-based peer assessment. In D. D. Williams, S. L. Howell, & M. Hricko (Eds.), Online assessment, measurement, and evaluation: Emerging practices (pp. 131−152). Hershey, PA: Information Science.
CR  - Matsuno, S. (2009). Self-, peer–, and teacher–assessments in Japanese university EFL writing classrooms. Language Testing, 26(1), 75−100. DOI: 10.1177/0265532208097337
CR  - Mehrdad, N., Bigdelib, S., & Ebrahimia, H. (2012). A comparative study on self, peer and teacher evaluation to evaluate clinical skills of nursing students. Procedia - Social and Behavioral Sciences, 47, 1847−1852.
CR  - McDonald, B. (2016). Peer assessment that works: A guide for teachers. London: Rowman & Littlefield.
CR  - McLeod, S. G., Brown, G. C., McDaniels, W., & Sledge, L. (2009). Improving writing with a PAL: harnessing the power of peer assisted learning with the reader’s assessment rubrics. International Journal of Teaching and Learning in Higher Education, 20(3), 488−502.
CR  - McNamara, T. F. (1996). Measuring second language performance. London and New York: Longman.
CR  - Moskal, B. M. (2000). Scoring rubrics: What, when and how? Practical Assessment, Research & Evaluation, 7(3), 1-5. Retrieved from: https://pareonline.net/getvn.asp?v=7&n=3
CR  - Moskal, B. M., & Leydens, J. A. (2000). Scoring rubric development: Validity and reliability. Practical Assessment, Research & Evaluation, 7(10), 1-6. Retrieved from http://pareonline.net/getvn.asp?v=7&n=10
CR  - Mutwarasibo, F. (2016). University students’ attitudes towards peer assessment and reactions to peer feedback on group writing. Rwanda Journal, Series A: Arts and Humanities, 1(1), 32−48. DOI: 10.4314/rj.v1i1.4A
CR  - Myford, C. M., & Wolfe, E. W. (2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of Applied Measurement, 5(2), 189−227.
CR  - Napoles, J. (2008). Relationships among instructor, peer, and self-evaluations of undergraduate music education majors' micro-teaching experiences. Journal of Research in Music Education, 56(1), 82−91. DOI: 10.1177/0022429408323071
CR  - Nitko, A.J. (2004). Educational assessment of students. Upper Saddle River, NJ: Pearson.
CR  - Ounis, M. (2017). A comparison between holistic and analytic assessment of speaking. Journal of Language Teaching and Research, 8(4), 679−690. DOI: 10.17507/jltr.0804.06
CR  - Petkov, D., & Petkova, O. (2006). Development of scoring rubrics for IS projects as an assessment tool. Issues in Informing Science and Information Technology, 3, 499−510. DOI: 10.28945/910
CR  - Prins, F. J., Sluijsmans, D. M. A., Kirschner, P. A., & Strijbos, J. W. (2005). Formative peer assessment in CSCL environment: a case study. Assessment & Evaluation in Higher Education, 30(4), 417−444. DOI: 10.1080/02602930500099219
CR  - Reddy, M. Y. (2010). Design and development of rubrics to improve assessment outcomes. A pilot study in a master's level business program in India. Quality Assurance in Education, 19(1), 84−104. DOI: 10.1108/09684881111107771
CR  - Sluijmans, D. M. A., Brand-Gruwel, S., van Merrienboer, J. J. G., & Bastiaens, T. J. (2003). The training of peer assessment skills to promote the development of reflections skills in teacher education. Studies in Educational Evaluation, 29(1), 23−42. DOI: 10.1016/S0191-491X(03)90003-4
CR  - Smith, H., Cooper, A., & Lancaster, L. (2002) Improving the quality of undergraduate peer assessment: A case for student and staff development. Innovations in Education and Teaching International, 39(1), 71−81. DOI: 10.1080/13558000110102904
CR  - Şahin, M. G., Taşdelen Teker, G. & Güler, N. (2016). An analysis of peer assessment through many facet Rasch model. Journal of Education and Practice, 7(32), 172−181.
CR  - Şahin, S. (2008). An application of peer assessment in higher education. The Turkish Online Journal of Educational Technology, 7(2), 5−10.
CR  - Şaşmaz Ören, F. (2018). Self, peer and teacher assessments: What is the level of relationship between them? European Journal of Education Studies, 4(7), 1−19. DOI: 10.5281/zenodo.1249959
CR  - Tan, Ş. (2015). Eğitimde ölçme ve değerlendirme KPSS el kitabı [Measurement and evaluation in education: KPSS handbook]. Ankara: Pegem.
CR  - Taşdelen Teker, G., Şahin, G., & Baytemir, K. (2016). Using generalizability theory to investigate the reliability of peer assessment. Journal of Human Sciences, 13(3), 5574−5586. DOI: 10.14687/jhs.v13i3.4155
CR  - Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68(3), 249−276. DOI: 10.3102/00346543068003249
CR  - Topping, K. J. (2009) Peer assessment. Theory into Practice, 48(1), 20−27. DOI: 10.1080/00405840802577569
CR  - Topping, K. J., Smith, E. F., Swanson, I., & Elliot, A. (2000). Formative peer assessment of academic writing between postgraduate students. Assessment & Evaluation in Higher Education, 25(2), 149−169. DOI: 10.1080/713611428
CR  - van den Berg, I., Admiraal, W., & Pilot, A. (2006). Designing student peer assessment in higher education: analysis of written and oral peer feedback. Teaching in Higher Education, 11(2), 135−147. DOI: 10.1080/13562510500527685
CR  - Wen, M. L., & Tsai, C. C. (2006). University students’ perceptions of and attitudes toward (online) peer assessment. Higher Education, 51(1), 27−44. DOI: 10.1007/s10734-004-6375-8
CR  - Wiseman, C. S. (2008). Investigating selected facets in measuring second language writing ability using holistic and analytic scoring methods (Unpublished doctoral dissertation). Columbia University. New York.
CR  - Wiseman, C. S. (2012). Comparison of the performance of analytic vs. holistic scoring rubrics to assess L2 writing. Iranian Journal of Language Testing, 2(1), 59−92.
CR  - Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8, 370–371.
CR  - Yakar, L. (2019). Tamamlayıcı ölçme ve değerlendirme teknikleri III [Complementary measurement and evaluation techniques]. N. Doğan (Eds.), in Eğitimde Ölçme ve Değerlendirme [Measurement and Evaluation in Education] (pp. 245−270). Ankara: Pegem.
CR  - Yune, S. J., Lee, S. Y., Im, S. J., Kam, B. S., & Baek, S. Y. (2018). Holistic rubric vs. analytic rubric for measuring clinical performance levels in medical students. BMC Medical Education, 18(1), 1−6. DOI: 10.1186/s12909-018-1228-9
UR  - https://doi.org/10.19128/turje.609073
L1  - https://dergipark.org.tr/en/download/article-file/832414
ER  -