Analyzing the Effects of Test, Student, and School Predictors on Science Achievement: An Explanatory IRT Modeling Approach

Year 2022, Volume: 13 Issue: 1, 40 - 53, 29.03.2022

Serap Büyükkıdık , Okan Bulut

https://doi.org/10.21031/epod.1013784

Cited By: 4

Abstract

This study aimed to investigate the impact of item features (i.e., content domain), student characteristics (i.e., gender), and school variables (i.e., school type) on students’ responses to a nationwide, large-scale assessment in Turkey. The sample consisted of 7507 students who participated in the 2016 administration of the Transition from Primary to Secondary Education Exam (TPSEE, referred to as “TEOG” in Turkey). Explanatory item response modeling was used for analyzing the effects of content domain, gender, school type, and their interactions on students’ responses to the science items on the exam. Five explanatory models were constructed to examine the effects of the item, student, and school variables sequentially. Results indicated that female students were more likely to answer the items correctly than male students. Also, students from private schools performed better than students from public schools. In terms of content, the biology items appeared to be significantly easier than the physics items. All interactions between the predictors were significant except for the Gender x School Type and Content x Gender x School Type interactions. The interactions between the predictors suggested that test developers, teachers, and stakeholders should be aware of potential item-level bias occurring in the science items due to complex interactions among the items, students, and schools characteristics.

Keywords

Explanatory item response modeling, science, school type, gender, content

References

• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
• Ayva Yörü, F. G., & Atar, H. Y. (2019). Determination of differential item functioning (DIF) according to SIBTEST, Lord’s [chi-squared], Raju’s area measurement and Breslow-Day methods. Journal of Pedagogical Research, 3(3), 139-150. https://doi.org/10.33902/jpr.v3i3.137
• Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. https://arxiv.org/pdf/1406.5823.pdf
• Bell, J. F. (2001). Investigating gender differences in the science performance of 16-year-old pupils in the UK. International Journal of Science Education, 23(5), 469-486. https://doi.org/10.1080/09500690120123
• Berberoğlu, G., & Kalender, İ. (2005). Öğrenci başarısının yıllara, okul türlerine, bölgelere göre incelenmesi: ÖSS ve PISA analizi. Journal of Educational Sciences & Practices, 4(7), 21-35. http://www.ebuline.com/english/pdfs/7_2.pdf
• Burkam, D. T., Lee, V. E., & Smerdon, B. A. (1997). Gender and science learning early in high school: Subject matter and laboratory experiences. American Educational Research Journal, 34(2), 297-331. https://doi.org/10.3102/00028312034002297
• Carnevale, A. P., Strohl, J., Ridley, N., & Gulish, A. (2018). Three educational pathways to good jobs: High school, middle skills, and bachelor’s degree. https://1gyhoq479ufd3yna29x7ubjn-wpengine.netdna-ssl.com/wp-content/uploads/3ways-FR.pdf
• Coleman, J. S., Hoffer, T., & Kilgore, S. (1982). Cognitive outcomes in public and private schools. Sociology of Education, 55(2-3), 65-76. https://www.jstor.org/stable/pdf/2112288.pdf?casa_token=2urqntYyuZQAAAAA:Rj2XcpFpD4Asklsmj_minXEoi7CsxMD1kg7yrb81rVd2wuN_j7zMZu6feBoRaNnN53xFobtwRojV4LkD8cv8WzWUgMWGGMbIBgaVwN2cMWCRp8G0VoI
• De Boeck, P., & Wilson, M. (Eds.) (2004). Explanatory item response models. Springer.
• Eriksson, K., Björnstjerna, M., & Vartanova, I. (2020). The relation between gender egalitarian values and gender differences in academic achievement. Frontiers in Psychology, 11. https://doi.org/10.3389/fpsyg.2020.00236
• Figlio, D.N., & Stone, J.A. (1997). School choice and student performance: Are private schools really better? (Discussion Paper 1141-97). Institute for Research on Poverty, University of Wisconsin-Madison.
• Gierl, M., Khaliq, S. N., & Boughton, K. (1999, June). Gender differential item functioning in mathematics and science: Prevalence and policy implications. In Annual Meeting of the Canadian Society for the Study of Education, Canada.
• Hambleton, R., Swaminathan, H., & Rogers, H. (1991). Fundamentals of item response theory. Sage.
• Hughes, G. (2001). Exploring the availability of student scientist identities within curriculum discourse: An anti-essentialist approach to gender-inclusive science. Gender & Education, 13(3), 275-290. https://doi.org/10.1080/09540250120063562
• Hyde, J. S. (2005). The gender similarities hypothesis. American Psychologist, 60(6), 581-592. https://doi.org/10.1037/0003-066X.60.6.581
• Hyde, J. S., & Linn, M. C. (2006). Gender similarities in mathematics and science. Science, 314(5799), 599-600. https://doi.org/10.1126/science.1132154
• International Association for the Evaluation of Educational Achievement. (1988). Science achievement in seventeen countries (A Preliminary Report). Pergamon Press.
• Kalaycioğlu, D. B., & Berberoğlu, G. (2010). Differential item functioning analysis of the science and mathematics items in the university entrance examinations in Turkey. Journal of Psychoeducational Assessment, 29(5), 467-478. https://doi.org/10.1177/0734282910391623
• Kan, A., Bulut, O., & Cormier, D. C. (2018). The impact of item stem format on the dimensional structure of mathematics assessments. Educational Assessment, 24(1), 13-32. https://doi.org/10.1080/10627197.2018.1545569
• Lee, V. E., & Burkam, D. T. (1996). Gender differences in middle grade science achievement: Subject domain, ability level, and course emphasis. Science Education, 80(6), 613-650. https://doi.org/10.1002/(SICI)1098-237X(199611)80:6<613::AID-SCE1>3.0.CO;2-M
• Legewie, J., & DiPrete, T. A. (2014). The high school environment and the gender gap in science and engineering. Sociology of Education, 87(4), 259-280. https://doi.org/10.1177/0038040714547770
• Liou, P. Y., & Bulut, O. (2020). The effects of item format and cognitive domain on students’ science performance in TIMSS 2011. Research in Science Education, 50(1), 99-121. https://doi.org/10.1007/s11165-017-9682-7
• Liu, O. L., & Wilson, M. (2009). Gender differences in large-scale math assessments: PISA trend 2000 and 2003. Applied Measurement in Education, 22(2), 164-184. https://doi.org/10.1080/08957340902754635
• Martinková P., & Drabinová A. (2018) ShinyItemAnalysis for teaching psychometrics and to enforce routine analysis of educational tests. The R Journal, 10(2), 503-515. https://doi.org/10.32614/RJ-2018-074
• Meinck, S., & Brese, F. (2019). Trends in gender gaps: Using 20 years of evidence from TIMSS. Large-Scale Assessments in Education, 7(1), 1-23. https://doi.org/10.1186/s40536-019-0076-3
• Ministry of National Education. (2015). Ortaöğretim kurumlarına geçiş uygulaması tercih ve yerleştirme e-kılavuzu 2015. http://odsgm.meb.gov.tr/meb_iys_dosyalar/2015_05/28024630_ekilavuz28.05.2015.pdf
• Mullis, I. V. S., Martin, M. O., Foy, P., Kelly, D. L., & Fishbein, B. (2020). TIMSS 2019 international results in mathematics and science. https://timssandpirls.bc.edu/timss2019/international-results/
• Newhouse, D., & Beegle, K. (2006). The effect of school type on academic achievement evidence from indonesia. Journal of Human Resources, 41(3), 529-557. https://doi.org/10.3368/jhr.XLI.3.529
• Ober, C., Loisel, D. A., & Gilad, Y. (2008). Sex-specific genetic architecture of human disease. Nature Reviews Genetics, 9(12), 911-922. https://doi.org/10.1038/nrg2415
• Organisation for Economic Co-operation and Development. (2016). PISA 2015 results: Excellence and equity in education (Vol. 1). OECD Publishing. https://doi.org/10.1787/9789264266490-en
• Organisation for Economic Co-operation and Development. (2021). Education at a Glance 2021: OECD indicators. OECD Publishing. https://doi.org/10.1787/b35a14e5-en
• Petscher, Y., Compton, D. L., Steacy, L., & Kinnon, H. (2020). Past perspectives and new opportunities for the explanatory item response model. Annals of Dyslexia, 70(2), 160-179. https://doi.org/10.1007/s11881-020-00204-y
• Quinn, D. M., & Cooc, N. (2015). Science achievement gaps by gender and race/ethnicity in elementary and middle school. Educational Researcher, 44(6), 336-346. https://doi.org/10.3102/0013189X15598539
• R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing.
• Reilly, D., Neumann, D. L., & Andrews, G. (2015). Sex differences in mathematics and science achievement: A meta-analysis of National Assessment of Educational Progress Assessments. Journal of Educational Psychology, 107(3), 645-662. https://doi.org/10.1037/edu0000012
• Rijmen, F., Tuerlinckx, F., De Boeck, P., & Kuppens, P. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8(2), 185-205. https://doi.org/10.1037/1082-989X.8.2.185
• Sinnes, A. T., & Løken, M. (2014). Gendered education in a gendered world: looking beyond cosmetic solutions to the gender gap in science. Cultural Studies of Science Education, 9(2), 343-364. https://doi.org/10.1007/s11422-012-9433-z
• Stoet, G., & Geary, D. C. (2018). The gender-equality paradox in science, technology, engineering, and mathematics education. Psychological Science, 29(4), 581-593. https://doi.org/10.1177/0956797617741719
• Wang, M.-T., Eccles, J. S., & Kenny, S. (2013). Not lack of ability but more choice: Individual and gender differences in choice of careers in science, technology, engineering, and mathematics. Psychological Science, 24(5), 770-775. https://doi.org/10.1177/0956797612458937
• Wilson, M., De Boeck, P., & Carstensen, C. H. (2008). Explanatory item response models: A brief introduction. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 91-120). Hogrefe.
• Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125-145. https://doi.org/10.1177/014662168400800201
• Young, D. J., & Fraser, B. J. (1994). Gender differences in science achievement: Do school effects make a difference? Journal of Research in Science Teaching, 31(8), 857-871. https://doi.org/10.1002/tea.3660310808
• Young, D. J., & Fraser, B. J. (1992, April). Sex differences in science achievement: A multilevel analysis. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco. https://files.eric.ed.gov/fulltext/ED356947.pdf
• Zhang, D., & Campbell, T. (2015). An examination of the impact of teacher quality and “opportunity gap” on student science achievement in China. International Journal of Science and Mathematics Education, 13(3), 489-513. https://doi.org/10.1007/s10763-013-9491-z