Examining The Rater Reliability of Laboratory Activities of Gifted Students In Science Courses With The Many Facet Rasch Model

Abstract

In this study, it was aimed to examine the inter-rater reliability of the same performance tasks scored by seven-raters with poster scoring rubrics, science-chemistry laboratory applications performance scoring rubrics, and open-ended achievement test scoring rubrics within the framework of the many facet Rasch model. The participants were 27 sixth grade gifted-students attending BİLSEM selected by criterion sampling from purposive sampling types and seven teachers serving as raters. Density Open-Ended Achievement Test, Poster Scoring rubrics and Science/Chemistry Laboratory Applications Process Performance Analytical Rubric were used to collect data of the study. In the study, FACETS was used for the MFRM, JASP-0.17.2.1 was used for model – data fit analyses and assumption tests. As a result, it was seen that individuals could be distinguished at the group level in terms of ability levels, the items differed in terms of difficulty levels and there was a difference between the raters in terms of severity/leniency. As a result, it was seen that individuals could be distinguished in terms of ability levels at the group level, the items differed in terms of difficulty levels and there was a difference between the raters in terms of severity/leniency except for the open-ended achievement test. It was seen that the scoring based on the three scoring tools distinguished the individuals in terms of ability levels and the items differed in terms of difficulty levels. In the scoring of the poster and experimental performance evaluation scoring tool, scorers numbered six and seven had scoring severity/leniency errors, while no scoring errors Halo effect in open-ended and doing experiment performance scoring were found in all scoring. There were no inconsistent scoring errors in the scoring of the open-ended achievement test, poster evaluation and experiment performances according to the scoring rubric of the density of irregularly shaped solids. Keywords: Many facet Rasch measurement model, gifted students, rater effects

Keywords

• Many facet Rasch measurement model
• gifted students
• rater effects

References

1. Akben, N., (2011). Developing a scientific inquiry supported laboratory course for prospective teachers. (Thesis number: 290633). [Doctoral dissertation, T. C. Gazi University]. HigherEducation Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
2. Aktaş, M. & Alıcı, D. (2017). Investigation of Rater Reliability in Checklist, Analytical Rubric and Rating Scales According to Generalisability Theory, International Journal Of Eurasia Social Sciences, Vol: 8, Issue: 29, pp. (991-1010). https://search.trdizin.gov.tr/tr/yayin/detay/263050/kontrol-listesi-analitik-rubrik-ve-dereceleme-olceklerinde-puanlayici-guvenirliginin-genellenebilirlik-kuramina-gore-incelenmesi
3. Altunova, N., Artun, H., 2019. Rubric Development Study for the Evaluation of Experiments in Science Laboratory Practices Course. 6th International Multidisciplinary Studies Congress.
4. Atılgan, 2004. A research on the comparison of generalisability theory and multivariate Rasch model.(Thesis number: 144251). [Doctoral dissertation, T. C. Hacettepe University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
5. Bacanak A., 2008. Investigation of the effectiveness of the web-based programme created for science and technology course performance evaluation forms.(Thesis number: 179206). [Doctoral dissertation, T. C. Karadeniz Technical University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
6. Baştürk, R. (2008). Applying the many facet Rasch model to evaluate PowerPoint presentation performance in higher education. Assessment and Evaluation in Higher Education, 33(4), 431-444. https://doi.org/10.1080/02602930701562775
7. Baştürk, R. & Işıkoğlu, N. (2008). Analysing the functional quality of pre-school education institutions with Many Facet rasch measurement model. Educational Sciences in Theory and Practice, 8(1), 7-32. https://search.trdizin.gov.tr/tr/yayin/detay/78688
8. Bıkmaz Bilgen, Ö., & Doğan, N. (2017). Comparison of Inter-rater Reliability Determination Techniques. Journal of Measurement and Evaluation in Education and Psychology, 8(1), 63-78. https://doi.org/10.21031/epod.294847

9. Brentari, E., & Golia, S. (2008). Measuring job satisfaction in the social services sector with the Rasch model. Journal of Applied Measurement, 9(1), 45-56. https://www.researchgate.net/publication/5668940_Measuring_job_satisfaction_in_the_social_services_sector_with_the_Rasch_Model
10. Cohen. J. R., Swerdlik M. E., & Phillips, S. M. (1996). Psychological testing and assessment. (3rd Ed.). London: Mayfield
11. Covill. A. E. (2012). College Students' Use of a Writing Rubric: Effect on Quality of Writing, Self-Efficacy, and Writing Practices. The Journal of Writing Assessment. (5)1. https://escholarship.org/uc/item/5f79g6kz
12. Çaylak, B., (2009). Investigation of science activities applied in science and art centres. (Thesis number: 246649). [Master's thesis, T. C. İnönü University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
13. Çetin, B., & İlhan, M. (2017). Investigation of rater rigour and generosity in open-ended mathematics questions scored with rubrics based on standard and SOLO Taxonomy. Education and Science, 42(189). http://dx.doi.org/10.15390/EB.2017.5082
14. Deliceoğlu, G., & Çıkrıkçı Demirtaşlı. N,. (2012). Comparison of the reliability of the rating scale for football skills based on generalisability and classical test theory. Hacettepe Journal of Sport Sciences. 23 (1),1-12. https://dergipark.org.tr/tr/download/article-file/151222
15. Esfandiari, R. (2015). Rater errors among peer-assessors: applying the many-facet Rasch measurement model. Iranian Journal of Applied Linguistics, 18(2), 77-107. https://doi.org/10.18869/acadpub.ijal.18.2.77
16. Farrokhi, F., & Esfandiari, R. (2011). A many-facet Rasch model to detect halo effect in three types of raters. Theory & Practice in Language Studies, 1(11), 1531-1540. https://doi.org/10.4304/tpls.1.11.1531-1540
17. Gisev, N., Bell, J. S., & Chen, T. F. (2013). Interrater agreement and interrater reliability: key concepts, approaches, and applications. Research in Social and Administrative Pharmacy, 9(3), 330-338. https://doi:10.1016/j.sapharm.2012.04.004
18. Göktürk, Ö., Uzun, N. B., & Aktaş, M. (2022). Performance scale development study for science-chemical laboratory applications. Online Science Education Journal, 7(2),72-84. https://dergipark.org.tr/en/download/article-file/2796015
19. Güler, N. (2008). Comparison of Reliability According to Generalisability Theory and Classical Test Theory on Random Data. Education and Science, 36 (162) https://egitimvebilim.ted.org.tr/index.php/EB/article/view/993
20. Hernández-Ureña, O., & Montero-Rojas, E. (2023). Validation of a standardised performance test for selection of Architecture students with the Many-Facet Rasch Measurement Model. Revista de Arquitectura (Bogotá), 25(1), 3-11. https://doi.org/10.14718/RevArq.2023.25.4040
21. Hung, S.P., Chen, P.S., & Chen, H.C. (2012). Improving creativity performance assessment: A rater effect examination with many facet Rasch model. Creativity Research Journal, 24(4), 345-357. https://doi.org/10.1080/10400419.2012.730331
22. İlhan, M. (2015). Investigation of rater effects in open-ended mathematics questions scored with rubrics based on standard and solo taxonomy with Many Facet rasch model.(Thesis number: 398394). [Doctoral dissertation, Gaziantep University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
23. Karakaya, İ. (2015). Comparison of self, peer and instructor assessments in the portfolio assessment by using many facet Rasch model. Journal of Education and Human Development, 4(2), 182-192. https://doi.org10.15640/jehd.v4n2a22
24. Linacre, J.M. (2014). A user's guide to FACETS Rasch-model computer programmes. Retrieved on 13.03.2024 from http://www.winsteps.com/a/facets-manual.pdf.
25. Linacre, J. M. (2018). A user's guide to Winsteps. Chicago, IL: Winsteps.com
26. Ministry of National Education General Directorate of Special Education and Guidance Services, Draft Framework Education Programme for Individuals with Special Abilities, (2014). http://orgm.meb.gov.tr/meb_iys_dosyalar/2014_01/27021830_ereveprogramtasla.pdf (Access Date: 12.01.2024).
27. MEB. (2022). 2022- 2023 science and art centres student identification guide. Retrieved from https://orgm.meb.gov.tr.
28. Myford, C.M., & Wolfe, E.W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386-422. https://pubmed.ncbi.nlm.nih.gov/14523257/
29. Myford, C. M., & Wolfe, E. W. (2003). Detecting and measuring rater effects using manyfacet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386-422. Retrieved from http://psycnet.apa.org/record/2003-09517-007.
30. Myford, C.M., & Wolfe, E.W. (2004). Detecting and measuring rater effects using manyfacet Rasch measurement: Part II. Journal of Applied Measurement, 5(2), 189-227. Retrieved from https://europepmc.org/abstract/med/15064538.
31. Nakamura, N. (2002). Teacher Assessment and Peer Assessment in Practice. Educational Studies. 44, 143, 204-215.
32. Özyer, K. K. (2018). Scoring open-ended questions in the open education system with the Many Facet Rasch Model analysis method. Journal of Open Education Applications and Research, 4(1), 61-77. https://dergipark.org.tr/en/download/article-file/419501
33. Sarıtaş, S., 2015. Investigation of Self, Peer and Teacher Scores in the Evaluation of Problem Solving Skills with Many Facet Rasch Measurement Model. (Thesis number: 429376). [Master's Thesis, Gazi University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
34. Schumacker, R. E. (1996). Many-Facet Rasch Model Selection Criteria: Examining Residuals and More. https://files.eric.ed.gov/fulltext/ED397117.pdf
35. Semerci, Ç. (2012). Evaluation of Students' Opinions on the Department of ITET According to Rasch Measurement Model (Fırat University Example). NWSA: Education Sciences, 7(2), 777-784. https://dergipark.org.tr/tr/pub/nwsaedu/issue/19816/211967
36. Semerci, Ç., Semerci, N., & Duman, B. (2013). Analysing graduate students' seminar presentation performances with Many FacetRasch model. Sakarya University Journal of Faculty of Education, 25, 7-22. https://dergipark.org.tr/tr/download/article-file/115660
37. Sudweeks, R. R., Reeve, S., & Bradshaw, W. S. (2004). A comparison of generalisability theory and many-facet Rasch measurement in an analysis of college sophomore writing. Assessing Writing, 9(3), 239-261. https://doi.org/10.1016/j.asw.2004.11.001
38. Şata, M., (2019). Investigation of the Effect of Rater Training on Rater Behaviours in Performance Evaluation Process (Thesis number: 626117). [Doctoral dissertation, Gazi University]. Higher Education Institution National Thesis Centre. AVAILABLE AT https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
39. Sata, M., & Karakaya, İ. (2021). Investigating the Effect of Rater Training on Differential Rater Function in Assessing Academic Writing Skills of Higher Education Students. Journal of Measurement and Evaluation in Education and Psychology, 12(2), 163-181. AVAILABLE AT https://doi.org/10.21031/epod.842094
40. Sata, M. & Karakaya, I. (2022). Investigating the impact of rater training on rater errors in the process of assessing writing skill. International journal of assessment tools in education, 9(2), 492-514. AVAILABLE AT https://doi.org/10.21449/ijate.877035
41. T.C. Ministry of National Education (MEB), General Directorate of Special Education and Guidance Services, 2022. Education of gifted individuals, strategy and implementation guide. https://orgm.meb.gov.tr/meb_iys_dosyalar/2013_11/25034903_zelyeteneklibireylerineitimistratejiveuygulamaklavuzu.pdf
42. Tekin D., (2023). Laboratory Activities and Application Performances of Students: Application Status of General Chemistry II Course at University (Thesis number: 813890). [Master's thesis, Çanakkale On Sekiz Mart University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
43. Tobaş C., (2020). Investigation of Differentiating Rater Behaviours in the Evaluation of Performance with Many Facet Rasch Measurement Model (Thesis number: 629923). [Master's thesis, Gazi University]. Higher Education Institution National Thesis Centre. https://tez.yok.gov.tr/UlusalTezMerkezi/giris.jsp
44. Ulukök, Ş., Çelik, H. & Sarı, U. (2013). The effect of computer-assisted applications related to simple electrical circuits on the development of experimental process skills. Journal of Theoretical Education Science, 6(1), 77-101. http://hdl.handle.net/11630/5434
45. Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement: Transactions of the Rasch Measurement SIG, 8(3), 370. https://rasch.org/rmt/rmt83.htm
46. Wu, S. M., & Tan, S. (2016). Managing rater effects through the use of FACETS analysis: the case of a university placement test. Higher Education Research & Development, 35(2), 380-394. https://doi.org/10.1080/07294360.2015.1087381
47. Yüzüak, A. V., Yüzüak, B., & Kaptan, F. (2015). Analysing the performance task in the direction of peer groups and teacher approaches with Many Facet rasch measurement model. Journal of Measurement and Evaluation in Education and Psychology, 6(1), 1-11. https://doi.org/10.21031/epod.57425