Examining The Rater Reliability of Laboratory Activities of Gifted Students In Science Courses With The Many Facet Rasch Model

Year 2025, Volume: 16 Issue: 4, 216 - 240, 31.12.2025

Başak Akar , Bayram Çetin

Abstract

In this study, it was aimed to examine the inter-rater reliability of the same performance tasks scored by seven-raters with poster scoring rubrics, science-chemistry laboratory applications performance scoring rubrics, and open-ended achievement test scoring rubrics within the framework of the many facet Rasch model. The participants were 27 sixth grade gifted-students attending BİLSEM selected by criterion sampling from purposive sampling types and seven teachers serving as raters. Density Open-Ended Achievement Test, Poster Scoring rubrics and Science/Chemistry Laboratory Applications Process Performance Analytical Rubric were used to collect data of the study. In the study, FACETS was used for the MFRM, JASP-0.17.2.1 was used for model – data fit analyses and assumption tests. As a result, it was seen that individuals could be distinguished at the group level in terms of ability levels, the items differed in terms of difficulty levels and there was a difference between the raters in terms of severity/leniency. As a result, it was seen that individuals could be distinguished in terms of ability levels at the group level, the items differed in terms of difficulty levels and there was a difference between the raters in terms of severity/leniency except for the open-ended achievement test. It was seen that the scoring based on the three scoring tools distinguished the individuals in terms of ability levels and the items differed in terms of difficulty levels. In the scoring of the poster and experimental performance evaluation scoring tool, scorers numbered six and seven had scoring severity/leniency errors, while no scoring errors Halo effect in open-ended and doing experiment performance scoring were found in all scoring. There were no inconsistent scoring errors in the scoring of the open-ended achievement test, poster evaluation and experiment performances according to the scoring rubric of the density of irregularly shaped solids.
Keywords: Many facet Rasch measurement model, gifted students, rater effects

Keywords

Many facet Rasch measurement model , gifted students , rater effects

Ethical Statement

• Çalışmamız özgün ve günceldir. • Yararlandığımız bütün çalışmalara eksiksiz ve doğru bir biçimde atıfta bulunduk ve alıntıları APA 7’ye göre yaptık. • Yazarlar arasında herhangi bir çıkar çatışması bulunmamaktadır. • Ham veri talep edildiğinde verebiliriz. • Çalışmamızı sadece söz konusu derginize sunduk. Tüm bu bilgilerin doğru olduğunu beyan ederiz.

Supporting Institution

-

Project Number

-

Thanks

-

References

• Akben, N., (2011). Developing a scientific inquiry supported laboratory course for prospective teachers. (Thesis number: 290633). [Doctoral dissertation, T. C. Gazi University]. HigherEducation Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
• Aktaş, M. & Alıcı, D. (2017). Investigation of Rater Reliability in Checklist, Analytical Rubric and Rating Scales According to Generalisability Theory, International Journal Of Eurasia Social Sciences, Vol: 8, Issue: 29, pp. (991-1010). https://search.trdizin.gov.tr/tr/yayin/detay/263050/kontrol-listesi-analitik-rubrik-ve-dereceleme-olceklerinde-puanlayici-guvenirliginin-genellenebilirlik-kuramina-gore-incelenmesi
• Altunova, N., Artun, H., 2019. Rubric Development Study for the Evaluation of Experiments in Science Laboratory Practices Course. 6th International Multidisciplinary Studies Congress.
• Atılgan, 2004. A research on the comparison of generalisability theory and multivariate Rasch model.(Thesis number: 144251). [Doctoral dissertation, T. C. Hacettepe University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
• Bacanak A., 2008. Investigation of the effectiveness of the web-based programme created for science and technology course performance evaluation forms.(Thesis number: 179206). [Doctoral dissertation, T. C. Karadeniz Technical University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
• Baştürk, R. (2008). Applying the many facet Rasch model to evaluate PowerPoint presentation performance in higher education. Assessment and Evaluation in Higher Education, 33(4), 431-444. https://doi.org/10.1080/02602930701562775
• Baştürk, R. & Işıkoğlu, N. (2008). Analysing the functional quality of pre-school education institutions with Many Facet rasch measurement model. Educational Sciences in Theory and Practice, 8(1), 7-32. https://search.trdizin.gov.tr/tr/yayin/detay/78688
• Bıkmaz Bilgen, Ö., & Doğan, N. (2017). Comparison of Inter-rater Reliability Determination Techniques. Journal of Measurement and Evaluation in Education and Psychology, 8(1), 63-78. https://doi.org/10.21031/epod.294847
• Brentari, E., & Golia, S. (2008). Measuring job satisfaction in the social services sector with the Rasch model. Journal of Applied Measurement, 9(1), 45-56. https://www.researchgate.net/publication/5668940_Measuring_job_satisfaction_in_the_social_services_sector_with_the_Rasch_Model
• Cohen. J. R., Swerdlik M. E., & Phillips, S. M. (1996). Psychological testing and assessment. (3rd Ed.). London: Mayfield
• Covill. A. E. (2012). College Students' Use of a Writing Rubric: Effect on Quality of Writing, Self-Efficacy, and Writing Practices. The Journal of Writing Assessment. (5)1. https://escholarship.org/uc/item/5f79g6kz
• Çaylak, B., (2009). Investigation of science activities applied in science and art centres. (Thesis number: 246649). [Master's thesis, T. C. İnönü University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
• Çetin, B., & İlhan, M. (2017). Investigation of rater rigour and generosity in open-ended mathematics questions scored with rubrics based on standard and SOLO Taxonomy. Education and Science, 42(189). http://dx.doi.org/10.15390/EB.2017.5082
• Deliceoğlu, G., & Çıkrıkçı Demirtaşlı. N,. (2012). Comparison of the reliability of the rating scale for football skills based on generalisability and classical test theory. Hacettepe Journal of Sport Sciences. 23 (1),1-12. https://dergipark.org.tr/tr/download/article-file/151222
• Esfandiari, R. (2015). Rater errors among peer-assessors: applying the many-facet Rasch measurement model. Iranian Journal of Applied Linguistics, 18(2), 77-107. https://doi.org/10.18869/acadpub.ijal.18.2.77
• Farrokhi, F., & Esfandiari, R. (2011). A many-facet Rasch model to detect halo effect in three types of raters. Theory & Practice in Language Studies, 1(11), 1531-1540. https://doi.org/10.4304/tpls.1.11.1531-1540
• Gisev, N., Bell, J. S., & Chen, T. F. (2013). Interrater agreement and interrater reliability: key concepts, approaches, and applications. Research in Social and Administrative Pharmacy, 9(3), 330-338. https://doi:10.1016/j.sapharm.2012.04.004
• Göktürk, Ö., Uzun, N. B., & Aktaş, M. (2022). Performance scale development study for science-chemical laboratory applications. Online Science Education Journal, 7(2),72-84. https://dergipark.org.tr/en/download/article-file/2796015
• Güler, N. (2008). Comparison of Reliability According to Generalisability Theory and Classical Test Theory on Random Data. Education and Science, 36 (162) https://egitimvebilim.ted.org.tr/index.php/EB/article/view/993
• Hernández-Ureña, O., & Montero-Rojas, E. (2023). Validation of a standardised performance test for selection of Architecture students with the Many-Facet Rasch Measurement Model. Revista de Arquitectura (Bogotá), 25(1), 3-11. https://doi.org/10.14718/RevArq.2023.25.4040
• Hung, S.P., Chen, P.S., & Chen, H.C. (2012). Improving creativity performance assessment: A rater effect examination with many facet Rasch model. Creativity Research Journal, 24(4), 345-357. https://doi.org/10.1080/10400419.2012.730331
• İlhan, M. (2015). Investigation of rater effects in open-ended mathematics questions scored with rubrics based on standard and solo taxonomy with Many Facet rasch model.(Thesis number: 398394). [Doctoral dissertation, Gaziantep University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
• Karakaya, İ. (2015). Comparison of self, peer and instructor assessments in the portfolio assessment by using many facet Rasch model. Journal of Education and Human Development, 4(2), 182-192. https://doi.org10.15640/jehd.v4n2a22
• Linacre, J.M. (2014). A user's guide to FACETS Rasch-model computer programmes. Retrieved on 13.03.2024 from http://www.winsteps.com/a/facets-manual.pdf.
• Linacre, J. M. (2018). A user's guide to Winsteps. Chicago, IL: Winsteps.com
• Ministry of National Education General Directorate of Special Education and Guidance Services, Draft Framework Education Programme for Individuals with Special Abilities, (2014). http://orgm.meb.gov.tr/meb_iys_dosyalar/2014_01/27021830_ereveprogramtasla.pdf (Access Date: 12.01.2024).
• MEB. (2022). 2022- 2023 science and art centres student identification guide. Retrieved from https://orgm.meb.gov.tr.
• Myford, C.M., & Wolfe, E.W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386-422. https://pubmed.ncbi.nlm.nih.gov/14523257/
• Myford, C. M., & Wolfe, E. W. (2003). Detecting and measuring rater effects using manyfacet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386-422. Retrieved from http://psycnet.apa.org/record/2003-09517-007.
• Myford, C.M., & Wolfe, E.W. (2004). Detecting and measuring rater effects using manyfacet Rasch measurement: Part II. Journal of Applied Measurement, 5(2), 189-227. Retrieved from https://europepmc.org/abstract/med/15064538.
• Nakamura, N. (2002). Teacher Assessment and Peer Assessment in Practice. Educational Studies. 44, 143, 204-215.
• Özyer, K. K. (2018). Scoring open-ended questions in the open education system with the Many Facet Rasch Model analysis method. Journal of Open Education Applications and Research, 4(1), 61-77. https://dergipark.org.tr/en/download/article-file/419501
• Sarıtaş, S., 2015. Investigation of Self, Peer and Teacher Scores in the Evaluation of Problem Solving Skills with Many Facet Rasch Measurement Model. (Thesis number: 429376). [Master's Thesis, Gazi University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
• Schumacker, R. E. (1996). Many-Facet Rasch Model Selection Criteria: Examining Residuals and More. https://files.eric.ed.gov/fulltext/ED397117.pdf
• Semerci, Ç. (2012). Evaluation of Students' Opinions on the Department of ITET According to Rasch Measurement Model (Fırat University Example). NWSA: Education Sciences, 7(2), 777-784. https://dergipark.org.tr/tr/pub/nwsaedu/issue/19816/211967
• Semerci, Ç., Semerci, N., & Duman, B. (2013). Analysing graduate students' seminar presentation performances with Many FacetRasch model. Sakarya University Journal of Faculty of Education, 25, 7-22. https://dergipark.org.tr/tr/download/article-file/115660
• Sudweeks, R. R., Reeve, S., & Bradshaw, W. S. (2004). A comparison of generalisability theory and many-facet Rasch measurement in an analysis of college sophomore writing. Assessing Writing, 9(3), 239-261. https://doi.org/10.1016/j.asw.2004.11.001
• Şata, M., (2019). Investigation of the Effect of Rater Training on Rater Behaviours in Performance Evaluation Process (Thesis number: 626117). [Doctoral dissertation, Gazi University]. Higher Education Institution National Thesis Centre. AVAILABLE AT https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
• Sata, M., & Karakaya, İ. (2021). Investigating the Effect of Rater Training on Differential Rater Function in Assessing Academic Writing Skills of Higher Education Students. Journal of Measurement and Evaluation in Education and Psychology, 12(2), 163-181. AVAILABLE AT https://doi.org/10.21031/epod.842094
• Sata, M. & Karakaya, I. (2022). Investigating the impact of rater training on rater errors in the process of assessing writing skill. International journal of assessment tools in education, 9(2), 492-514. AVAILABLE AT https://doi.org/10.21449/ijate.877035
• T.C. Ministry of National Education (MEB), General Directorate of Special Education and Guidance Services, 2022. Education of gifted individuals, strategy and implementation guide. https://orgm.meb.gov.tr/meb_iys_dosyalar/2013_11/25034903_zelyeteneklibireylerineitimistratejiveuygulamaklavuzu.pdf
• Tekin D., (2023). Laboratory Activities and Application Performances of Students: Application Status of General Chemistry II Course at University (Thesis number: 813890). [Master's thesis, Çanakkale On Sekiz Mart University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
• Tobaş C., (2020). Investigation of Differentiating Rater Behaviours in the Evaluation of Performance with Many Facet Rasch Measurement Model (Thesis number: 629923). [Master's thesis, Gazi University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
• Ulukök, Ş., Çelik, H. & Sarı, U. (2013). The effect of computer-assisted applications related to simple electrical circuits on the development of experimental process skills. Journal of Theoretical Education Science, 6(1), 77-101. http://hdl.handle.net/11630/5434
• Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement: Transactions of the Rasch Measurement SIG, 8(3), 370. https://rasch.org/rmt/rmt83.htm
• Wu, S. M., & Tan, S. (2016). Managing rater effects through the use of FACETS analysis: the case of a university placement test. Higher Education Research & Development, 35(2), 380-394. https://doi.org/10.1080/07294360.2015.1087381
• Yüzüak, A. V., Yüzüak, B., & Kaptan, F. (2015). Analysing the performance task in the direction of peer groups and teacher approaches with Many Facet rasch measurement model. Journal of Measurement and Evaluation in Education and Psychology, 6(1), 1-11. https://doi.org/10.21031/epod.57425