TY  - JOUR
T1  - Analyzing the Effects of Test, Student, and School Predictors on Science Achievement: An Explanatory IRT Modeling Approach
AU  - Büyükkıdık, Serap
AU  - Bulut, Okan
PY  - 2022
DA  - March
Y2  - 2022
DO  - 10.21031/epod.1013784
JF  - Journal of Measurement and Evaluation in Education and Psychology
JO  - JMEEP
PB  - Association for Measurement and Evaluation in Education and Psychology
WT  - DergiPark
SN  - 1309-6575
SP  - 40
EP  - 53
VL  - 13
IS  - 1
LA  - en
AB  - This study aimed to investigate the impact of item features (i.e., content domain), student characteristics (i.e., gender), and school variables (i.e., school type) on students’ responses to a nationwide, large-scale assessment in Turkey. The sample consisted of 7507 students who participated in the 2016 administration of the Transition from Primary to Secondary Education Exam (TPSEE, referred to as “TEOG” in Turkey). Explanatory item response modeling was used for analyzing the effects of content domain, gender, school type, and their interactions on students’ responses to the science items on the exam. Five explanatory models were constructed to examine the effects of the item, student, and school variables sequentially. Results indicated that female students were more likely to answer the items correctly than male students. Also, students from private schools performed better than students from public schools. In terms of content, the biology items appeared to be significantly easier than the physics items. All interactions between the predictors were significant except for the Gender x School Type and Content x Gender x School Type interactions. The interactions between the predictors suggested that test developers, teachers, and stakeholders should be aware of potential item-level bias occurring in the science items due to complex interactions among the items, students, and schools characteristics.
KW  - Explanatory item response modeling
KW  - science
KW  - school type
KW  - gender
KW  - content
CR  - American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
CR  - Ayva Yörü, F. G., & Atar, H. Y. (2019). Determination of differential item functioning (DIF) according to SIBTEST, Lord’s [chi-squared], Raju’s area measurement and Breslow-Day methods. Journal of Pedagogical Research, 3(3), 139-150. https://doi.org/10.33902/jpr.v3i3.137
CR  - Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. https://arxiv.org/pdf/1406.5823.pdf
CR  - Bell, J. F. (2001). Investigating gender differences in the science performance of 16-year-old pupils in the UK. International Journal of Science Education, 23(5), 469-486. https://doi.org/10.1080/09500690120123
CR  - Berberoğlu, G., & Kalender, İ. (2005). Öğrenci başarısının yıllara, okul türlerine, bölgelere göre incelenmesi: ÖSS ve PISA analizi. Journal of Educational Sciences & Practices, 4(7), 21-35. http://www.ebuline.com/english/pdfs/7_2.pdf
CR  - Burkam, D. T., Lee, V. E., & Smerdon, B. A. (1997). Gender and science learning early in high school: Subject matter and laboratory experiences. American Educational Research Journal, 34(2), 297-331. https://doi.org/10.3102/00028312034002297
CR  - Carnevale, A. P., Strohl, J., Ridley, N., & Gulish, A. (2018). Three educational pathways to good jobs: High school, middle skills, and bachelor’s degree. https://1gyhoq479ufd3yna29x7ubjn-wpengine.netdna-ssl.com/wp-content/uploads/3ways-FR.pdf
CR  - Coleman, J. S., Hoffer, T., & Kilgore, S. (1982). Cognitive outcomes in public and private schools. Sociology of Education, 55(2-3), 65-76. https://www.jstor.org/stable/pdf/2112288.pdf?casa_token=2urqntYyuZQAAAAA:Rj2XcpFpD4Asklsmj_minXEoi7CsxMD1kg7yrb81rVd2wuN_j7zMZu6feBoRaNnN53xFobtwRojV4LkD8cv8WzWUgMWGGMbIBgaVwN2cMWCRp8G0VoI
CR  - De Boeck, P., & Wilson, M. (Eds.) (2004). Explanatory item response models. Springer.
CR  - Eriksson, K., Björnstjerna, M., & Vartanova, I. (2020). The relation between gender egalitarian values and gender differences in academic achievement. Frontiers in Psychology, 11. https://doi.org/10.3389/fpsyg.2020.00236
CR  - Figlio, D.N., & Stone, J.A. (1997). School choice and student performance: Are private schools really better? (Discussion Paper 1141-97). Institute for Research on Poverty, University of Wisconsin-Madison.
CR  - Gierl, M., Khaliq, S. N., & Boughton, K. (1999, June). Gender differential item functioning in mathematics and science: Prevalence and policy implications. In Annual Meeting of the Canadian Society for the Study of Education, Canada.
CR  - Hambleton, R., Swaminathan, H., & Rogers, H. (1991). Fundamentals of item response theory. Sage.
CR  - Hughes, G. (2001). Exploring the availability of student scientist identities within curriculum discourse: An anti-essentialist approach to gender-inclusive science. Gender & Education, 13(3), 275-290. https://doi.org/10.1080/09540250120063562
CR  - Hyde, J. S. (2005). The gender similarities hypothesis. American Psychologist, 60(6), 581-592. https://doi.org/10.1037/0003-066X.60.6.581
CR  - Hyde, J. S., & Linn, M. C. (2006). Gender similarities in mathematics and science. Science, 314(5799), 599-600. https://doi.org/10.1126/science.1132154
CR  - International Association for the Evaluation of Educational Achievement. (1988). Science achievement in seventeen countries (A Preliminary Report). Pergamon Press.
CR  - Kalaycioğlu, D. B., & Berberoğlu, G. (2010). Differential item functioning analysis of the science and mathematics items in the university entrance examinations in Turkey. Journal of Psychoeducational Assessment, 29(5), 467-478. https://doi.org/10.1177/0734282910391623
CR  - Kan, A., Bulut, O., & Cormier, D. C. (2018). The impact of item stem format on the dimensional structure of mathematics assessments. Educational Assessment, 24(1), 13-32. https://doi.org/10.1080/10627197.2018.1545569
CR  - Lee, V. E., & Burkam, D. T. (1996). Gender differences in middle grade science achievement: Subject domain, ability level, and course emphasis. Science Education, 80(6), 613-650. https://doi.org/10.1002/(SICI)1098-237X(199611)80:6 <613::AID-SCE1>3.0.CO;2-M
CR  - Legewie, J., & DiPrete, T. A. (2014). The high school environment and the gender gap in science and engineering. Sociology of Education, 87(4), 259-280. https://doi.org/10.1177/0038040714547770
CR  - Liou, P. Y., & Bulut, O. (2020). The effects of item format and cognitive domain on students’ science performance in TIMSS 2011. Research in Science Education, 50(1), 99-121. https://doi.org/10.1007/s11165-017-9682-7
CR  - Liu, O. L., & Wilson, M. (2009). Gender differences in large-scale math assessments: PISA trend 2000 and 2003. Applied Measurement in Education, 22(2), 164-184. https://doi.org/10.1080/08957340902754635
CR  - Martinková P., & Drabinová A. (2018) ShinyItemAnalysis for teaching psychometrics and to enforce routine analysis of educational tests. The R Journal, 10(2), 503-515. https://doi.org/10.32614/RJ-2018-074
CR  - Meinck, S., & Brese, F. (2019). Trends in gender gaps: Using 20 years of evidence from TIMSS. Large-Scale Assessments in Education, 7(1), 1-23. https://doi.org/10.1186/s40536-019-0076-3
CR  - Ministry of National Education. (2015). Ortaöğretim kurumlarına geçiş uygulaması tercih ve yerleştirme e-kılavuzu 2015. http://odsgm.meb.gov.tr/meb_iys_dosyalar/2015_05/28024630_ekilavuz28.05.2015.pdf
CR  - Mullis, I. V. S., Martin, M. O., Foy, P., Kelly, D. L., & Fishbein, B. (2020). TIMSS 2019 international results in mathematics and science. https://timssandpirls.bc.edu/timss2019/international-results/
CR  - Newhouse, D., & Beegle, K. (2006). The effect of school type on academic achievement evidence from indonesia. Journal of Human Resources, 41(3), 529-557. https://doi.org/10.3368/jhr.XLI.3.529
CR  - Ober, C., Loisel, D. A., & Gilad, Y. (2008). Sex-specific genetic architecture of human disease. Nature Reviews Genetics, 9(12), 911-922. https://doi.org/10.1038/nrg2415
CR  - Organisation for Economic Co-operation and Development. (2016). PISA 2015 results: Excellence and equity in education (Vol. 1). OECD Publishing. https://doi.org/10.1787/9789264266490-en
CR  - Organisation for Economic Co-operation and Development. (2021). Education at a Glance 2021: OECD indicators. OECD Publishing. https://doi.org/10.1787/b35a14e5-en
CR  - Petscher, Y., Compton, D. L., Steacy, L., & Kinnon, H. (2020). Past perspectives and new opportunities for the explanatory item response model. Annals of Dyslexia, 70(2), 160-179. https://doi.org/10.1007/s11881-020-00204-y
CR  - Quinn, D. M., & Cooc, N. (2015). Science achievement gaps by gender and race/ethnicity in elementary and middle school. Educational Researcher, 44(6), 336-346. https://doi.org/10.3102/0013189X15598539
CR  - R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing.
CR  - Reilly, D., Neumann, D. L., & Andrews, G. (2015). Sex differences in mathematics and science achievement: A meta-analysis of National Assessment of Educational Progress Assessments. Journal of Educational Psychology, 107(3), 645-662. https://doi.org/10.1037/edu0000012
CR  - Rijmen, F., Tuerlinckx, F., De Boeck, P., & Kuppens, P. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8(2), 185-205. https://doi.org/10.1037/1082-989X.8.2.185
CR  - Sinnes, A. T., & Løken, M. (2014). Gendered education in a gendered world: looking beyond cosmetic solutions to the gender gap in science. Cultural Studies of Science Education, 9(2), 343-364. https://doi.org/10.1007/s11422-012-9433-z
CR  - Stoet, G., & Geary, D. C. (2018). The gender-equality paradox in science, technology, engineering, and mathematics education. Psychological Science, 29(4), 581-593. https://doi.org/10.1177/0956797617741719
CR  - Wang, M.-T., Eccles, J. S., & Kenny, S. (2013). Not lack of ability but more choice: Individual and gender differences in choice of careers in science, technology, engineering, and mathematics. Psychological Science, 24(5), 770-775. https://doi.org/10.1177/0956797612458937
CR  - Wilson, M., De Boeck, P., & Carstensen, C. H. (2008). Explanatory item response models: A brief introduction. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 91-120). Hogrefe.
CR  - Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125-145. https://doi.org/10.1177/014662168400800201
CR  - Young, D. J., & Fraser, B. J. (1994). Gender differences in science achievement: Do school effects make a difference? Journal of Research in Science Teaching, 31(8), 857-871. https://doi.org/10.1002/tea.3660310808
CR  - Young, D. J., & Fraser, B. J. (1992, April). Sex differences in science achievement: A multilevel analysis. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco. https://files.eric.ed.gov/fulltext/ED356947.pdf
CR  - Zhang, D., & Campbell, T. (2015). An examination of the impact of teacher quality and “opportunity gap” on student science achievement in China. International Journal of Science and Mathematics Education, 13(3), 489-513. https://doi.org/10.1007/s10763-013-9491-z
UR  - https://doi.org/10.21031/epod.1013784
L1  - https://dergipark.org.tr/en/download/article-file/2042611
ER  -