Examination of Common Exams Held by Measurement and Assessment Centers: Many Facet Rasch Analysis

Year 2021, , 649 - 666, 05.09.2021

Gülden Kaya Uyanık , Tuğba Demirtaş Tolaman , Duygu Gür Erdoğan

https://doi.org/10.21449/ijate.730977

Abstract

This paper aims to examine and assess the questions included in the “Turkish Common Exam” for sixth graders held in the first semester of 2018 which is one of the common exams carried out by The Measurement and Evaluation Centers, in terms of question structure, quality and taxonomic value. To this end, the test questions were examined by three specialists with expertise in different fields in terms of structure, content, and taxonomic values. The test questions were then rated by raters with expertise in different fields according to the criteria set by the researchers. Hence, the study employed the descriptive survey model. The data obtained from the assessment of the questions were analyzed using the Many Facet Rasch Model (MFRM). According to the findings, of the 20 questions included in the exam, 5 (five) are in the category of “Remembering”, 12 (twelve) in the category of “Understanding”, 2 (two) in the category of “Analyzing” and 1 (one) in the category of “Evaluating.” Accordingly, the number of questions that measure higher-order thinking skills was lower than the number of lower-level questions. In addition, the study contained three facets: raters, tasks (items), and criteria. There were no differences among the raters (a Turkish Education Specialist, a Program Development Specialist, and a Testing and Assessment Specialist) in terms of severity and leniency: all the raters were in agreement. Finally, in this study, the questions met the criteria measuring the structural features, while they failed to meet the criteria measuring the quality and clarity.

Keywords

Turkish Course, Common Exam, Many Facet Rasch Analysis, Multiple-Choice Item

References

• Aktaş, E. (2017). Öğretmen adaylarının farklı metin türlerine yönelik soru sorma becerilerinin Yenilenmiş Bloom Taksonomisine göre değerlendirilmesi [Evaluation of the questioning skills of teachers candidates towards the different text types accordıng to The Renewed Bloom Taxonomy]. Electronic Turkish Studies, 12(25), 99 118. http://dx.doi.org/10.7827/TurkishStudies.12274
• Akyol, H., Yıldırım, K., Seyit, A., & Çetinkaya, Ç. (2013). Anlamaya yönelik nasıl sorular soruyoruz? [What kinds of questions do we ask for making meaning?]. Mersin University Journal of Education, 9(1), 41-56.
• Anderson, L. W. (2005). Objectives, evaluation, and the improvement of education, Studies in Education Evaluation, 31, 102-113. https://doi.org/10.1016/j.stueduc.2005.05.004
• Anderson, L.W., Krathwohl, D.R., Airaisan, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., Raths, J., & Wittrock, M.C. (2001). A taxonomy for learning, teaching and assessing: a revision of Bloom’s Taxonomy of educational objectives. Addison Wesley Longman, Inc.
• Andre, T. (1979). Does Answering higher-level questions while reading facilitate productive learning? Review of Educational Research, 49(2), 280-318.
• Anıl, D. (2009). Uluslararası öğrenci başarılarını değerlendirme programı (PISA)’nda Türkiye’deki öğrencilerin Fen Bilimleri başarılarını etkileyen faktörler [Factors effecting Science achievement of science students in Programme for International Students’ Achievement (PISA) in Turkey]. Education and Science, 34(152), 87-100.
• Anshel, M.H., Weatherby, N.L., Kang M. & Watson, T. (2009). Rasch calibration of a unidimensional perfectionism ınventory for sport. Psychology of Sport and Exercise, 10(2009), 210-216. https://doi.org/10.1016/j.psychsport.2008.07.006
• Ateş, S. (2011). Evaluatıon of Fifth-Grade Turkish Course Learnıng and Teaching Process in Terms of Comprehensıon Instruction [Unpublished doctoral dissertation, Gazi University]. Gazi University Libraries.
• Ateş, S., Güray, E., Döğmeci, Y., & Gürsoy, F. F. (2016). Öğretmen ve öğrenci sorularının gerektirdikleri zihinsel süreçler açısından karşılaştırılması [Comparison of questions of teachers and students in terms of level]. Research in Reading and Writing Instruction, 4(1), 1-13.
• Baştürk, R. (2009). Applying The Many – Facet Rasch Model to evaluate powerpoint presentation performance ın higher education. Assesment And Evaluation in Higher Education, 33(4), 431-444. https://doi.org/10.1080/02602930701562775
• Bircan, E. (2012). Türkçe öğretmeni adaylarının hazırladığı soruların yeniden yapılandırılan Bloom Taksonomisine göre değerlendirilmesi [Evaluation of the questions prepared by Turkish language teacher candidates according to The Revised Bloom’s Taxonomy]. Kastamonu University Journal of Education, 20(3), 965-982.
• Bloom, B. (1956). Taxonomy of educational objectives. David Mckay. https://www.uky.edu/~rsand1/china2018/texts/Bloom%20et%20al%20 Taxonomy%20of%20Educational%20Objectives.pdf
• Bozkurt, B.Ü., Uzun, G.L., & Lee, Y. (2015). Korece ve Türkçe ders kitaplarındaki metin sonu sorularının karşılaştırılması: PISA 2009 sonuçlarına dönük bir tartışma [A comparıson of reading comprehension questions in Korean and Turkish textbooks: A discussion on PISA 2009 results]. International Journal of Language Academy, 3(4), 295-313. http://dx.doi.org/10.18033/ijla.327
• Brinthaupt, T.M., & Kang, M. (2012). Many-faceted rasch calibraton: an example using the self-talk scale. Assessment, 21(2), 241-249. https://doi.org/10.1177/1073191112446653
• Büyüköztürk, Ş. (2016). Sınavlar üzerine düşünceler [Thoughts on exams]. Kalem International Journal of Education and Human Sciences, 6(2), 345-356.
• Case, S.M. & Swanson, D.B. (2002). Constructing written test questions for the basic and clinical sciences. 3rd Ed (rev.) National Board of Medical Examiners.
• Cayhan, C., & Akın, E.(2015). TEOG sınavı Türkçe dersi sorularının Türkçe Dersi Öğretim Programındaki kazanımlar açısından değerlendirilmesi [The evaluation of Turkish lesson questions TEOG examinatıon in terms of Turkish lesson education program objectives]. Siirt University Journal of Social Sciences Institute, 5, 106-114.
• Chiavaroli, N. (2017). Negatively-worded multiple choice questions: An avoidable threat to validity. Practical Assessment, Research, and Evaluation, 22(1), 3. https://doi.org/10.7275/ca7y-mm27
• Çeçen, M. A., & Kurnaz, H. (2015). Ortaokul Türkçe dersi öğrenci çalışma kitaplarındaki tema değerlendirme soruları üzerine bir araştırma [Student workbook of secondary school Turkish course: A research on theme evaluation questions]. Journal of Karadeniz Social Sciences, 7(2).
• Çeliktürk Sezgin, Z., & Gedikoğlu Özilhan, Y. G. (2019). 1.-8. sınıf Türkçe ders kitaplarındaki metne dayalı anlama sorularının incelenmesi [Examining text-based comprehension questions in Turkish textbooks of the 1st- the 8st graders]. Journal of Mother Tongue Education, 7(2), 353-367. https://doi.org/10.16916/aded.530191
• Çepni, S., Özsevgenç, T. & Gökdere, M. (2003). Bilişsel gelişim ve formal operasyon dönem özelliklerine göre ÖSS fizik ve lise fizik sorularının incelenmesi [Examination of SSE physics and high school physics questions according to cognitive development and formal operation period features]. Journal of National Education, 157, 30-39.
• Çer, E. (2018). A comparison of mother-tongue curricula of successful countries in PISA and Turkey by higher-order thinking processes. Eurasian Journal of Educational Research, 73, 95-112.
• Day, R. R., & Park, J. (2005). Developing reading comprehension questions. Reading in a Foreign Language, 17(1), 60-73.
• Demiral, H., & Menşan, N. (2017). Sekizinci sınıf Türkçe dersinin PISA okuma becerilerine göre değerlendirilmesi [Evaluation of the eighth grade Turkish lesson according to PISA reading skills]. Küreselleşen dünyada eğitim (Edt: Özcan Demirel, Serkan Dinçer). Pegem Yayıncılık.
• Dillon, J.T. (2006). Effect of questions in education and other enterprises. In Westbury, I.& Milburn, G. (Eds.), Rethinking Schooling (pp.145 174). Routledge. https://doi.org/10.4324/9780203963180
• Downing, S. M. (2006). Twelve steps for effective test development. In Downing, S.M,& Haladyna, T. M. (Eds.), Handbook of test development, (pp.3-25). Routledge.
• Durukan, E. (2009). 7. sınıf Türkçe ders kitaplarındaki metinleri anlamaya yönelik sorular üzerine taksonomik bir inceleme [A taxonomic study on questions to understand texts in 7th grade Turkish Textbooks]. Journal of National Education, 181, 84-93.
• Eckes, T. (2011). Introduction to many-facet rasch measurement. Frankfurt Am.
• Engelhard, G. (1994). Examining rater errors in the assessment of written composition with a many-faceted rasch model. Journal of Educational Measurement. 31(2), 93-112. https://doi.org/10.1111/j.1745-3984.1994.tb00436.x
• Erdoğan, T. (2017). İlkokul dördüncü sınıf öğrencilerinin ve öğretmenlerinin Türkçe dersine ilişkin sordukları soruların Yenilenmiş Bloom Taksonomisi açısından görünümü [The view of primary school fourth grade students and teachers’ questions about Turkish language lessons in the terms of The Revised Bloom Taxonomy]. Education and Science, 42(192). http://dx.doi.org/10.15390/EB.2017.7407
• Eroğlu, D., & Kuzu, T.S. (2014). Türkçe ders kitaplarındaki dilbilgisi kazanımlarının ve sorularının Yenilenmiş Bloom Taksonomisine göre değerlendirilmesi [The evaluation of the grammar a cquisitions and questions in Turkish course books with respect to New Bloom Taxonomy]. Başkent University Journal of Education, 1(1), 72-80.
• Garden, R. A., & Orpwood, G. (1996). Development of The TIMSS Achievement Tests. Third International Mathematics and Science Study. Technical Report, 1.
• Govindasamy, P., del Carmen Salazar, M., Lerner, J., & Green, K. E. (2019). Assessing the reliability of the framework for equitable and effective teaching with the many-facet rasch model. Frontiers in Psychology, 10, 1363. https://doi.org/10.3389/fpsyg.2019.01363
• Göçer, A. (2008). İlköğretim Türkçe ders kitaplarinin ölçme ve değerlendirme açisindan incelenmesi [Analysis of Turkish course books for measurement and evaluation]. Atatürk University Journal of Social Sciences Institute, 11(1), 197-210.
• Göçer, A. (2016). Lisansüstü eğitim gören Türkçe öğretmenlerinin yazılı sınav sorularının incelenmesi [Investigation of written exam questions of Turkish teachers who upper graduate education]. Uşak University Journal of Social Sciences Institute, 9(27/3), 22-37.
• Güfta, H., & Zorbaz, K. Z. (2008). İlköğretim ikinci kademe türkçe dersi yazılı sınav sorularının düzeyleri üzerine bir değerlendirme [A review regarding levels of written examination questions for Turkish courses of the secondary school]. Çukurova University Journal of Social Sciences Institute, 17(2), 205-218.
• Güneş, F. (2007). Türkçe öğretimi ve zihinsel yapılandırma [Turkish teaching and mental structuring]. Nobel Yayın Dağıtım.
• Güneş, F. (2011). Dil Öğretim yaklaşımları ve Türkçe öğretimindeki uygulamalar [Language teaching approaches and their applications in teaching Turkısh]. Mustafa Kemal University Journal of Social Sciences Institute, 8(15), 123-148.
• Güneş, F. (2012). Testlerden etkinliklere türkçe öğretimi [Teaching Turkish from tests to activities]. Journal of Language and Literature, 1(1), 31-42.
• Haladyna, T.M., & Downing, S.M. (1989a). A taxonomy of multiple-choice item writing rules. Applied Measurement in Education, 2(1), 37 50. https://doi.org/10.1207/s15324818ame0201_3
• Haladyna, T.M., & Downing, S.M. (1989b). Validity of a taxonomy of multiple-choice item writing rules. Applied Measurement in Education, 2(1), 51 78. https://doi.org/10.1207/s15324818ame0201_4
• Haladayna, T.M. (1997). Writing test ıtems to evaluate higher order thinking. Viacom Company.
• Hetherman, S.C. (2004). An Application of Multi Faceted Rasch Measurement to Monitör Effectiveness of the Written Composition in English in The New York City Department of Education [Doctoral dissertation, Colombia University]. Colombia University Libraries.
• Hill, J.B. (2016). Questioning techniques: A study of instructional practice. Peabody Journal of Education, 91(5), 660-671. https://doi.org/10.1080/0161956X.2016.1227190
• Karadüz, A. (2010). Dil becerileri ve eleştirel düşünme [Language skills and the critical thinking]. Turkish Studies, 5(3), 1566 1593. http://dx.doi.org/10.7827/TurkishStudies.1572
• Karasar, N. (1998). Araştırmalarda rapor hazırlama yöntemi [Research Report Preparation Method]. Ankara: Pars Matbaacılık Sanayi.
• Kardeş-Birinci, D. (2014). Merkezi sistem ortak sınavlarında ilk deneyim: Matematik dersi [The first experıence in central system common exams: mathematics]. Journal of Research in Education and Teaching, 3(2), 8-16.
• Kavruk, H., & Çeçen, M.A. (2013). Türkçe dersi yazılı sınav sorularının bilişsel alan basamakları açısından değerlendirilmesi [Evaluation of Turkish language class exam questions in point of cognitive field levels]. Journal of Mother Tongue Education, 1(4), 1-9. https://doi.org/10.16916/aded.15990
• Kim, Y., Park, I., & Kang, M. (2012). Examining rater effects of the TGMD-2 on children with intellectual disability. Adapted Physical Activity Quarterly, 29, 346-365.
• Kocaarslan, M., & Yamaç, A. (2018). Sınıf öğretmenlerinin Türkçe dersi sınavlarında sordukları metne dayalı anlama sorularının incelenmesi [Investigating text-based comprehension questions primary school teachers ask in exams of Turkish course]. Trakya Journal of Education, 8(2), 431-448. https://doi.org/10.24315/trkefd.356769
• Kurudayıoğlu, M., & Çetin, Ö. (2015). Temel beceriler ve Türkçe öğretimi [Basic skills and Turkish education]. Journal of Mother Tongue Education, 3(3), 1 19. https://doi.org/10.16916/aded.65619
• Lane, S., Raymond, M.R., & Haladyna, T.M. (2015). Handbook of test development. Routledge.
• Linacre, J.M. (1989). Many-facet Rasch Measurement [Doctoral dissertation, University of Chicago]. Chicago University Libraries.
• Linacre, J. M. (2007). A User’s Guide to FACETS: Rasch Model Computer Programs. Chicago, IL.
• Maden, S. (2011). Türkçe dersi öğretmenlerinin ölçme değerlendirmeye ilişkin algıları [Turkish course teachers’ perceptions on measurement and evaluation]. Journal of National Education, 41(190), 212-233.
• Marzano, R.J., & Kendall, J.S. (2007). The new taxonomy of educational objectives. Sage.
• McMillan, J. H. (2013). Classroom assessment: principles and practice for effective standards-based instruction (6th Edition). Pearson.
• Mislevy, R.J., & Riconscente, M.M. (2006). Handbook of test development. In Downing, S.M,&Haladyna,T.M. (Eds.), Evidence-centered assessment design (pp.61-90). Routledge.
• Myford, C. M., & Wolfe, E. W. (2004). Detecting and measuring rater effects using Many-Facet Rasch measurement: part II. Journal of Applied Measurement, 5(2), 189-227.
• Noddings, N. (2018). Philosophy of education. Routledge.
• Önalan, K., & Nesrin, Z. (2015). Türkçe ders kitaplarındaki metin altı soruların aşamalı sınıflandırmaya göre incelenmesi [Examining text-based comprehension questions in turkish textbooks of the 1 st - the 8 st Graders]. International Journal of Languages’ Education and Teaching, 1527-1533.
• Özçelik, D.A. (2009). Test hazırlama klavuzu [Test Guide]. (4. Baskı). Pegem Akademi.
• Özdemir, M., Özdemir, O., & Çetinkaya, Ç. (2007, 15-17 November). Analysis of the questions in the primary Turkish course workbooks [Conference presentation]. 1. Ulusal İlköğretim Kongresi, Ankara.
• Peterson, D.S., & Taylor, B.M. (2012). Using higher order questioning to accelerate students’ growth in reading. The Reading Teacher, 65(5), 295 304. https://doi.org/10.1002/TRTR.01045
• Republic of Turkey Ministry of National Education (2006). Turkish Course Curriculum.
• Republic of Turkey Ministry of National Education (2019). Turkish Course Curriculum. http://mufredat.meb.gov.tr/ProgramDetay.aspx?PID=663
• Republic of Turkey Ministry of National Education (2020). Turkish Language Exam in Four Skills. https://www.meb.gov.tr/meb_iys_dosyalar/2020_01/20094146_Dort_Beceride_Turkce_Dil_Sinavi_Ocak_2020.pdf
• Sarar Kuzu, T. (2013). Türkçe ders kitaplarındaki metin altı sorularının Yenilenmiş Bloom Taksonomisindeki hatırlama ve anlama bilişsel düzeyleri açısından incelenmesi. [Investigation of the text following questions in Turkish course books with respect to their remembering and understanding cognition levels of The Revised Bloom Taxonomy]. Sivas Cumhuriyet University Faculty of Letters Journal of Social Sciences, 37(1), 58-76.
• Shaunessy, E. (2000). Questioning techniques in the gifted classroom. Gifted Child Today, 23(5), 14-21. https://doi.org/10.4219/gct-2000-752
• Stenner, A. J. (1990). Objectivity: Specific and General. Rasch Measurement Transactions, 3(4), 111.
• Topçu, E. (2017). TEOG Tarih sorularının Yenilenmiş Bloom Taksonomisine göre analizi [Analysis of History questions asked in the transition from primary to secondary education according to The Renewed Bloom Taxonomy]. International Journal of Turkish Education Sciences, 9, 321-335.
• Turkish Course Common Exam. (2018). The Measurement and Evaluation Center. http://sakarya.odm.meb.gov.tr
• Uto, M. (2020). Accuracy of performance-test linking based on a many-facet Rasch model. Behavior Research Methods, 1-15.https://doi.org/10.3758/s13428-020-01498-x
• Webb, N.L. (2007). Issue related to judging the alignmnet of curriculum standards and assessments. Applied Measurement in Education, 20(1), 7-25.
• Wilen, W.W. (1991). Questioning skills for teachers. National Education Association.
• Wright, B. D., & Linacre, J. M. (1994). Reasonable Mean-Square Fit Values. Rasch Measurement: Transactions of the Rasch Measurement SIG, 8(3), 370.
• Yeşilyurt, E. (2012). Öğretmen adaylarının bilişsel alanla ilgili sınama durumu soruları yazma yeterliklerinin değerlendirilmesi [Evaluating teacher candidates’ competencıes on writıng testing situation questions related to cognitive domain]. Kastamonu University Journal of Education, 20(2), 519-530.
• Yıldırım, K. (2012). Öğretmenlerin öğrencilerin okuduğunu anlama becerilerini değerlendirmede kullanabilecekleri bir sistem: Barrett Taksonomisi [A system to be used by teachers to evaluate students' reading comprehensıon skills: Barrett Taxonomy]. Mustafa Kemal University Journal of Social Sciences Institute, 9(18), 45-58.
• Yıldız, D. Ç. (2015). Türkçe dersi sınav sorularının yeniden yapılandırılan Bloom Taksonomisine göre analizi [The analysis of Turkish course exam questions according to re-constructed Bloom’s Taxonomy]. Gaziantep University Journal of Social Sciences, 14(2), 479-497.