Reliability of Ratings of Multidimensional Fluency Scale with Many-Facet Rasch Model

Year 2022, Volume: 9 Issue: 2, 470 - 491, 26.06.2022

Çiğdem Akın Arıkan , Pınar Kanık Uysal , Huzeyfe Bilge , Kasım Yıldırım

https://doi.org/10.21449/ijate.974214

Cited By: 1

Abstract

The aim of this study was to determine whether the reliability of raters was provided by assessing reading prosody using the Multidimensional Fluency Scale (MDFS). The study was completed with a cross-sectional design, and in line with this, the prosodic reading skills of 41 fifth-grade students were rated by elementary school classroom teachers and Turkish language arts teachers using the MDFS. Data obtained from the ratings were analyzed with the many-facet Rasch model (MFRM). When the findings are investigated, the reading prosody rubric used in the research served the purposes of the reading prosody criteria, the sub-dimensions of the rubric could be reliably differentiated, the determined criteria were reliable, and the criteria categories appear to be adequate. Additionally, the severity and leniency of raters were found to differ, and Turkish language arts teachers were found to perform more severe ratings than classroom teachers. It was found that raters were ranked reliably in terms of severity/leniency, and that their levels of severity/leniency differed from each other. Another result obtained is that the prosody criterion that students completed with the most difficulty was phrasing. Therefore, it was concluded that the MDFS is a reliable rubric and that researchers and teachers can reliably use it to assess prosodic reading skills.

Keywords

Rater bias, Rubric, Reading fluency, Prosody, Many-facet Rasch model, Rater reliability

References

• Acar, M., & Anıl, D. (2009). Sınıf öğretmenlerinin performans değerlendirme sürecindeki değerlendirme yöntemlerini kullanabilme yeterlilikleri, karşılaştıkları sorunlar ve çözüm önerileri [Classroom teacher evaluation methods to use in the performance assessment process qualification of able, they comparison problems and solution proposals]. Journal of TUBAV Science, 2(3), 354-363.
• American Psychological Associaton. (2015). Halo effect. In APA dictionary of psychology (2nd ed., p. 667).
• Allington, R.L. (1983). Fluency: the neglected reading goal. The Reading Teacher, 36(6), 556-561.
• Armut, M., & Türkyılmaz, M. (2017). Ortaokul öğrencilerinin okuma becerileri üzerine bir inceleme [An investigation on reading skills of middle school students]. Erzincan University Journal of Education Faculty, 20(1), 217 236. https://doi.org/10.17556/erziefd.330587
• Aşıkcan, M. (2019). Üçüncü sınıf öğrencilerinin akıcı okuma becerilerinin geliştirilmesine yönelik bir eylem araştırması [An action research on ımproving fluent reading skills of third-grade primary school students] [Unpublished doctoral dissertation]. Necmettin Erbakan University.
• Baird, J.A., Hayes, M., Johnson, R., Johnson, S., & Lamprianou, I. (2013). Marker effects and examination reliability. A Comparative exploration from the perspectives of generalisability theory, Rash model and multilevel modelling. Oxford: University of Oxford for Educational Assessment. Retrieved from http://dera.ioe.ac.uk/17683/1/2013-01-21-marker-effects-and-examinationreliability.pdf
• Barkaoui, K. (2008). Effects of scoring method and rater experience on ESL essay rating processes and outcomes [Doctoral Dissertation, Available from ProOuest Dissertations and Theses database]. UMI No: 304360302.
• Barrett, S., (2001). The impact of training on rater variability. International Education Journal, 2(1), 49-58.
• Baştuğ, M., (2021). Akıcı okumayı geliştirme: kavramlar, uygulamalar, değerlendirmeler [Developing reading fluency: concepts, practices, assessments]. Pegem Akademi Yayıncılık
• Baştuğ, M., & Keskin, H.K. (2012). Akıcı okuma becerileri ile anlama düzeyleri (basit ve çıkarımsal) arasındaki ilişki [The relationship between fluent reading skills and comprehension level (literal and ınferential)]. Ahi Evran University Journal of Kırşehir Education Faculty, 13(3), 227-244.
• Baştürk, R., & Işıkoglu, N. (2008). Analyzing process quality of early childhood education with many facet rash measurement model. Educational Sciences:Theory and Practice, 8(1), 25-32.
• Benjamin, R.G., & Schwanenflugel, P.J. (2010). Text complexity and oral reading prosody in young readers. Reading Research Quarterly, 45(4), 388 404. https://doi.org/10.1598/RRQ.45.4.2
• Benjamin, R.G., Schwanenflugel, P.J., Meisinger, E.B., Groff, C., Kuhn, M.R., & Steiner, L. (2013). A spectrographically grounded scale for evaluating reading expressiveness. Reading Research Quarterly, 48(2), 105-133. https://doi.org/10.1002/rrq.43
• Benzer, A., & Eldem, E. (2013). Türkçe ve edebiyat öğretmenlerinin ölçme ve değerlendirme araçları hakkında bilgi düzeyleri [Level of the information about Turkish and literature teachers’ measurement and assessment materials]. Kastamonu Education Journal, 21(2), 649-664.
• Bilge, H. (2019). Okuma, yazma ve konuşma akıcılık ile okuduğunu anlama ve kelime hazinesi arasındaki ilişki [The relationships between reading, writing and speaking fluencies, reading comprehension and vocabulary] [Unpublished doctoral dissertation]. Gazi University.
• Bond, T.G., & Fox, C.M. (2015). Applying the Rasch Model: Fundamental Measurement in the Human Sciences (3th ed.). Routledge.
• Brookhart, S. M. (2013). How to Create and Use Rubrics for Formative Assessment and Grading. ASCD.
• Buck, J., & Torgesen, J. (2003). The relationship between performance on a measure of oral reading fluency and performance on Florida comprehensive assessment test. FCRR Technical Report# 1. Florida Center for Reading Research.
• Ceyhan, S. (2019). Etkileşimli sesli okumanın öğrencilerin okuduğunu anlama, okuma motivasyonu ve akıcı okumalarına etkisi [The effect of interactive reading aloud on the reading comprehension, reading motivation and reading fluency of students] [Unpublished doctoral dissertation]. Gazi University.
• Coşkun, E., & Coşkun, H. (2014). İlkokul ve ortaokullardaki bitişik eğik yazı uygulamalarına ilişkin öğretmen, öğrenci ve veli görüşleri [teachers’, students’ and parents’ views on cursive italic handwriting]. Mustafa Kemal University Journal of Social Sciences Institute, 11(26), 209-223.
• Couper-Kuhlen, E. (1986). An Introduction to English Prosody. Edward Arnold.
• Crystal, D. (2008). A Dictionary of Linguistics and Phonetics (6th ed.). Blackwell Publishing.
• Çetinkaya, Ç., Ateş, S., & Yıldırım, K. (2016). Prozodik okumanın aracılık etkisi: Lise düzeyinde okuduğunu anlama ve akıcı okuma arasındaki ilişkilerin incelenmesi [The mediation effect of reading prosody: exploring the relations between reading fluency and reading comprehension at high school level]. Turkish Studies, 11(3). https://doi.org/10.7827/TurkishStudies.9339
• Daane, M.C., Campbell, J.R., Grigg, W.S., Goodman, M.J., & Oranje, A. (2005). The nation's report card: fourth-grade students reading aloud: NAEP 2002 special study of oral reading. Washington, D.C.: U.S. Department of Education. Institute of Education Sciences, National Center for Education Statistics.
• Doğan, B. (2013). Türkçe ve sınıf öğretmenlerinin okuma güçlüğüne ilişkin bilgileri ve okuma güçlüğü olan öğrencileri belirleyebilme düzeyleri [Determining Turkish language and elementary classroom teachers’ knowledge on dyslexia and their awareness of diagnosing students with dyslexia]. Research in Reading & Writing Instruction, 1(1), 20-33.
• Dowhower, S.L. (1991). Speaking of prosody: fluency's unattended bedfellow. Theory Into Practice, 30(3), 165-175. https://doi.org/10.1080/00405849109543497
• Eckes, T. (2005). Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly: An International Journal, 2(3), 197-221. https://doi.org/10.1207/s15434311laq0203_2
• Eckes, T. (2015). Introduction to Many-Facet Rasch Measurement: Analyzing and Evaluating Rater Mediated Assessments (Vol. 22). Peter Lang Edition. https://doi.org/10.1080/15366367.2018.1516094
• Engelhard, G. (2002). Monitoring raters in performance assessments. In G. A. Tindal & T. M. Haladyna (Eds.), Large scale assesment for all students: Validity, technical adequacy, and implementation (pp. 261-287). Erlbaum.
• Engelhard, G., & Myford, C.M. (2003). Monitoring faculty consultant performance in the advanced placement English Literature and composition program with a many‐faceted Rasch model. ETS Research Report Series, 2003(1), i-60.
• Erguvan, İ.D., & Dünya, B.A. (2020). Analyzing rater severity in a freshman composition course using many facet Rasch measurement. Language Testing in Asia, 10(1), 1-20. https://doi.org/10.1186/s40468-020-0098-3
• Esmer, B. (2019). Okuduğunu anlama ile akıcı okuma, okur benlik algısı, okumaya adanmışlık ve okuyucu tepkisi ilişkileri [Direct and inferential relations among reading comprehension, silent and oral reading fluency, reading self-concept, reading engagement and response to picturebooks] [Unpublished doctoral dissertation]. Gazi University.
• Fraenkel, J.R., & Wallen, N.E. (2009). How to Design and Evaluate Research in Education (7th ed.). McGraw-Hill.
• Godde, E., Bailly, G., Escudero, D., Bosse, M.L., & Gillet-Perret, E. (2017). Evaluation of reading performance of primary school children: objective measurements vs. subjective ratings.WOCCI 2017-6th Workshop on Child Computer Interaction, Nov 2017, Glasgow, United Kingtom.
• Godde, E., Bosse, M.L., & Bailly, G. (2019). A review of reading prosody acquisition and development. Reading and Writing, 33(2), 399-426. https://doi.org/10.1007/s11145-019-09968-1
• Goodwin, S., (2016). A many-facet Rasch analysis comparing essay rater behavior on an academic English reading/writing test used for two purposes. Assessing Writing, 30, 21-31. https://doi.org/10.1016/j.asw.2016.07.004
• Grosjean, F., & Collins, M. (1979). Breathing, pausing and reading. Phonetica, 36(2), 98-114. https://doi.org/10.1159/000259950
• Haiyang, S. (2010). An application of classical test theory and many facet Rasch measurement in analyzing the reliability of an English test for non-English major graduates. Chinese Journal of Applied Linguistics, 33(2), 87-102.
• Hallman, J. (2009). Reading aloud: comprehending, not word calling. In R. Stone (Ed.), Best practices for teaching reading: what award-winning classroom teachers do (pp. 39-43). Corwin Press.
• Haskins, T., & Aleccia, V. (2014). Toward a reliable measure of prosody: an investigation of rater consistency. International Journal of Education and Social Science, 1(5), 102-112.
• İlhan, M. (2015). Standart ve solo taksonomisine dayalı rubrikler ile puanlanan açık uçlu matematik sorularında puanlayıcı etkilerinin çok yüzeyli rasch modeli ile incelenmesi [The identification of rater effects on open-ended math questions rated through standard rubrics and rubrics based on the SOLO taxonomy in reference to the many facet Rasch model] [Unpublished doctoral dissertation]. Gaziantep University.
• Kanık Uysal, P., & Duman, A. (2020). The effects of fluency-oriented reading instruction on reading skills. Pegem Journal of Education and Instruction, 10(4), 1111–1146. https://doi.org/10.14527/pegegog.2020.034
• Kaya Tosun, D. (2019). Okuma çemberlerinin okuduğunu anlama, akıcı okuma, okuma motivasyonu ve sosyal beceriler üzerindeki etkisi ve okur tepkilerinin belirlenmesi [The effect of literature circles on reading comprehension, reading fluency, reading motivation and social skills and exploring of reader responses] [Unpublished doctoral dissertation]. Gazi University.
• Kaya Uyanık, G., Güler, N., Taşdelen Teker, G., & Demir, S. (2019). The analysis of elementary science education course activities through many-facet Rasch model. Kastamonu Education Journal, 27(1), 139-150. https://doi.org/10.24106/kefdergi.2417
• Keskin, H.K. (2012). Akıcı okuma yöntemlerinin okuma becerileri üzerindeki etkisi [Impact of reading fluency methods on reading skills] [Unpublished doctoral dissertation]. Gazi University.
• Kızıltaş, Y. (2019). Ana dili farklı ilkokul öğrencilerinin akıcı okuma ve okuduğunu anlama becerilerinin incelenmesi [The study of reading fluency and reading comprehension skills of primary school whose mother tongue is different] [Unpublished doctoral dissertation]. Gazi University.
• Köse, İ.A., Usta, H.G., & Yandı, A. (2016). Sunum yapma becerilerinin çok yüzeyli Rasch analizi ile değerlendirilmesi [Evaluation of presentation skills by using many facets rasch model]. Abant İzzet Baysal University Journal of Faculty of Education, 16(4), 1853-1864.
• Kuhn, M.R. (2007). Effective oral reading assessment (or why round robin reading doesn’t cut ıt). In J.R. Paratore & R.L. McCormack (Ed.), Classroom literacy assessment: making sense of what students know and do (pp. 101-112). The Guilford Press.
• Kuhn, M.R., & Stahl, S.A. (2013). Fluency: developmental and remedial practices-revisited. In D.E. Alvermann, N.J. Unrau, & R.B. Ruddell (Ed.), Theoretical models and processes of reading (pp. 385-412). International Reading Association.
• Li, G., Pan, Y., & Wang, W. (2021). Using generalizability theory and many-facet Rasch model to evaluate in-basket tests for managerial positions. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.660553
• Linacre, J.M. (1993). Generalizability theory and many-Facet Rasch measurement, in Paper presented at the Annual Meeting of the American Educational Research Association (Atlanta, GA).
• Linacre, J.M. (1994). Many-Facet Rasch Measurement. Chicago: Mesa Press.
• Linacre, J.M. (2002). What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions, 16(2), 878.
• Linacre, J.M. (2003). Size vs. significance: Infit and outfit mean-square and standardized chi-square fit statistic. Rasch Measurement Transactions, 17, 918.
• Linacre, J.M. (2011). Facets computer program for Many-Facet Rasch Measurement. https://www.winsteps.com/facets.htm#:~:text=Facets%20is%20designed%20to%20handle,further%20measurement%20and%20structural%20facets
• Linacre, J.M. (2012). Many facet Rasch measurement: Facets tutorials. http://www.winsteps.com/tutorials.htm
• Linacre, J.M. (2014). A user's guide to FACETS Rasch-model computer programs. http://www.winsteps.com/a/facets-manual.pdf
• Mathson, D.V., Allington, R.L., & Solic, K.L. (2006). Hijacking fluency and instructionally informative assessments. In T. Rasinski, C. Blachowicz, & K. Lems (Eds.), Fluency instruction: research-based best practices (pp. 106-119). The Guilford Press.
• McMillan, J.H. (2017). Classroom Assessment. Principles and Practice that Enhance Student Learning and Motivation (7th ed.), Pearson.
• MoNE. (2016). 5. Sınıf Türkçe ortaokul ders kitabı [Turkish secondary school textbook for 5th graders]. Devlet Kitapları.
• Morrison, T.G., & Wilcox, B. (2020). Assessing expressive oral reading fluency. Education Sciences, 10(59). https://doi.org/10.3390/educsci10030059
• Moser, G.P., Sudweeks, R.R., Morrison, T.G., & Wilcox, B. (2014). Reliability of ratings of children’s expressive reading. Reading Psychology, 35(1), 58 79. https://doi.org/10.1080/02702711.2012.675417
• Myford, C.M., & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386-422.
• Myford, C.M., & Wolfe, E.W. (2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of Applied Measurement, 5(2), 189-227.
• National Reading Panel. (2000). Teaching children to read: an evidence-based assessment of the scientific research litearature on reading and its implications for reading instruction: reports of the subgroups. Washington, DC: National Institute of Child Health Human Development.
• Overstreet, T.B. (2014). The effect of prosody instruction on reading fluency and comprehension among third-grade students [Unpublished doctoral dissertation]. Andrews University.
• Özbaşı, D. & Kumandaş-Öztürk, H. (2021). Öğretim materyallerinin çok yüzeyli Rasch analiziyle değerlendirilmesi [Evaluatıon of teaching materials with many-facet Rasch analysis]. Trakya Journal of Education, 11(1), 187-200.
• Paige, D.D., Smith, G., Rupley, W., & Wells, W. (2021). Reducing high-attaining readers to middling: the consequences of inadequate foundational skills instruction in a high-ses district. Literacy Research and Instruction, 60(1), 81 106. https://doi.org/10.1080/19388071.2020.1780653
• Palmer, M.L. (2010). The relationship between reading fluency, writing fluency, and reading comprehension ın suburban third-grade students [Unpublished doctoral dissertation]. San Diego State University.
• Rasinski, T. (2004). Assessing Reading Fluency. Honolulu, Hawai: Pasific Resources for Education and Learning.
• Rasinski, T. (2010). The Fluent Reader. Scholastic.
• Rasinski, T., Paige, D.D., Rains, C., Stewart, F., Julovich, B., Prenkert, D., Rupley, W.H., & Nicholas, W.D. (2017). Effects of intensive fluency instruction on the reading proficiency of third-grade struggling readers. Reading and Writing Quarterly, 33(6), 519-532. https://doi.org/10.1080/10573569.2016.1250144
• Rasinski, T., Rikli, A., & Johnston, S. (2009). Reading fluency: More than automaticity? More than a concern for the primary grades?. Literacy Research and Instruction, 48(4), 350-361. https://doi.org/10.1080/19388070802468715
• Samuels, S.J. (2006). Reading fluency:ıts past, present, and future. In T. Rasinski, C. Blachowicz, & K. Lems (Ed.), Fluency ınstruction research-based best practices (pp. 7-20). The Guildgord Press.
• Saracoğlu, S., Dedebali, N.C., Dinçer, B., & Dursun, F. (2011). Sınıf, fen ve teknoloji ile Türkçe öğretmenlerinin öğretmen stillerinin incelenmesi [Investigatıon of teaching styles of primary school, science and techonology, Turkish teachers]. Education Sciences, 6(3), 2313-2327.
• Schreiber, P.A. (1991). Understanding prosody's role in reading acquisition. Theory into Practice, 30(3), 158-164. https://doi.org/10.1080/00405849109543496
• Schumacker, R.E., & Smith, E.V. (2007). A Rasch perspective. Educational and Psychological Measurement, 67(3), 394-409. https://doi.org/10.1177/0013164406294776
• Schwanenflugel, P.J., Hamilton, A., Kuhn, M.R., Wisenbaker, J.M., & Stahl, S.A. (2004). Becoming a fluent reader: reading skill and prosodic features in the oral reading of young readers. Journal of Educational Psychology, 96(1), 119 129. https://doi.org/10.1037/0022-0663.96.1.119
• Shrout, P.E., & Fleiss, J.L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428. https://doi.org/10.1037/0033-2909.86.2.420
• Sinambela, S.E. (2017). Prosody as a tool for assessing reading fluency of adult esl students. Advances in Language and Literary Studies, 8(6), 83 87. https://doi.org/10.7575/aiac.alls.v.8n.6p.83
• Smith, G.S., & Paige, D.D. (2019). A study of reliability across multiple raters when using the NAEP and MDFS rubrics to measure oral reading fluency. Reading Psychology, 40(1), 34-69. https://doi.org/10.1080/02702711.2018.1555361
• Spafford, C.S., Pesce, A.J.I., & Grosser, G.S. (Ed.). (1998). The Cyclopedic Education Dictionary. Delmar Publishers.
• Stevens, D.D., & Levi, A.J. (2005). Introduction to Rubrics: an Assessment Tool to Save Grading Time, Convey Effective Feedback, and Promote Student Learning. Stylus.
• Sudweeks, R.R., Reeve, S., & Bradshaw, W. S. (2004). A comparison of generalizability theory and many-facet Rasch measurement in an analysis of college sophomore writing. Assessing Writing, 9(3), 239-261. https://doi.org/10.1016/j.asw.2004.11.001
• Şata, M., & Karakaya, İ. (2020). Investigation of the use of electronic portfolios in the determination of student achievement in higher education using the many-facet Rasch measurement model. Educational Policy Analysis and Strategic Research, 15(7-21).
• Taşkaya, S.M., & Muşta, M.C. (2008). Sınıf öğretmenlerinin Türkçe öğretim yöntemlerine ilişkin görüşleri [Teachers’ opinions on Turkish teaching methods]. Elektronik Sosyal Bilimler Dergisi, 7(25), 240-251.
• U.S. Department of Education. (2002). National assessment of educational progress (NAEP) 2002 oral reading fluency study. Washington, DC: Institute of Education Sciences, National Center for Education Statistics.
• Ulusoy, M., Ertem, İ.S., & Dedeoğlu, H. (2011). Evaluating pre-service teachers’ oral reading records prepared for the grades 1-5 considering the prosodic competences. Gazi University Journal of Gazi Educational Faculty, 31(3), 759-774.
• Valencia, S.W., Smith, A.T., Reece, A.M., Li, M., Wixson, K.K., & Newman, H. (2010). Oral reading fluency assessment: Issues of construct, criterion, and consequential validity. Reading Research Quarterly, 45, 270-291. https://doi.org/10.1598/RRQ.45.3.1
• VandenBos, G.R. (2015). APA Dictionary of Psychology. American Psychological Association.
• Xu, Y., & Liu, F. (2012). Intrinsic coherence of prosodic and segmental aspects of speech. In O. Niebuhr (Ed.), Understanding prosody: the role of context, function and communcation (Vol. 13, pp. 1-26). De Gruyter.
• Yan, X. (2014). An examination of rater performance on a local oral English proficiency test: A mixed methods approach. Language Testing, 31(4), 501 527. https://doi.org/10.1177/0265532214536171
• Yıldız, M., Kanık Uysal, P., Bilge, H., Wolters, A.P., Saka, Y., Yıldırım, K., & Rasinski, T. (2019). Relationship between Turkish eighth-grade students' oral reading efficacy, reading comprehension and achievement scores on a high-stakes achievement test. Reading Psychology, 40(4), 1-21. https://doi.org/10.1080/02702711.2018.1555363
• Yıldız, M., Yıldırım, K., Ateş, S., & Çetinkaya, Ç. (2009). An evaluation of the oral reading fluency of 4th graders with respect to prosodic characteristic. International Journal of Human Sciences, 6(1), 353-360.
• Young, C., & Rasinski, T. (2009). Implementing readers theatre as an approach to classroom fluency instruction. The Reading Teacher, 63(1), 4-13. https://doi.org/10.1598/RT.63.1.1
• Yüzüak, A., Yüzüak, B., & Kaptan, F. (2015). Performans görevinin akran gruplar ve öğretmen yaklaşımları doğrultusunda çok-yüzeyli Rasch ölçme modeli ile analizi [A many-facet Rasch measurement approach to analyze peer and teacher assessment for authentic assessment task]. Journal of Measurement and Evaluation in Education and Psychology, 6(1), 1-11.
• Zimmerman, B.S., Rasinski, T., Was, C.A., Rawson, K.A., Dunlosky, J., Kruse, S.D., & Nikbakht, E. (2019). Enhancing outcomes for struggling readers: Empirical analysis of the fluency development lesson. Reading Psychology, 40(1), 70 94. https://doi.org/10.1080/02702711.2018.1555365
• Zutell, J., & Rasinski, T. (1991). Training teachers to attend to their students' oral reading fluency. Theory Into Practice, 30(3), 211 217. https://doi.org/10.1080/00405849109543502