Analyzing the Effects of Test, Student, and School Predictors on Science Achievement: An Explanatory IRT Modeling Approach

Abstract

This study aimed to investigate the impact of item features (i.e., content domain), student characteristics (i.e., gender), and school variables (i.e., school type) on students’ responses to a nationwide, large-scale assessment in Turkey. The sample consisted of 7507 students who participated in the 2016 administration of the Transition from Primary to Secondary Education Exam (TPSEE, referred to as “TEOG” in Turkey). Explanatory item response modeling was used for analyzing the effects of content domain, gender, school type, and their interactions on students’ responses to the science items on the exam. Five explanatory models were constructed to examine the effects of the item, student, and school variables sequentially. Results indicated that female students were more likely to answer the items correctly than male students. Also, students from private schools performed better than students from public schools. In terms of content, the biology items appeared to be significantly easier than the physics items. All interactions between the predictors were significant except for the Gender x School Type and Content x Gender x School Type interactions. The interactions between the predictors suggested that test developers, teachers, and stakeholders should be aware of potential item-level bias occurring in the science items due to complex interactions among the items, students, and schools characteristics.

Keywords

• Explanatory item response modeling
• science
• school type
• gender
• content

References

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
2. Ayva Yörü, F. G., & Atar, H. Y. (2019). Determination of differential item functioning (DIF) according to SIBTEST, Lord’s [chi-squared], Raju’s area measurement and Breslow-Day methods. Journal of Pedagogical Research, 3(3), 139-150. https://doi.org/10.33902/jpr.v3i3.137
3. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1-48. https://arxiv.org/pdf/1406.5823.pdf
4. Bell, J. F. (2001). Investigating gender differences in the science performance of 16-year-old pupils in the UK. International Journal of Science Education, 23(5), 469-486. https://doi.org/10.1080/09500690120123
5. Berberoğlu, G., & Kalender, İ. (2005). Öğrenci başarısının yıllara, okul türlerine, bölgelere göre incelenmesi: ÖSS ve PISA analizi. Journal of Educational Sciences & Practices, 4(7), 21-35. http://www.ebuline.com/english/pdfs/7_2.pdf
6. Burkam, D. T., Lee, V. E., & Smerdon, B. A. (1997). Gender and science learning early in high school: Subject matter and laboratory experiences. American Educational Research Journal, 34(2), 297-331. https://doi.org/10.3102/00028312034002297
7. Carnevale, A. P., Strohl, J., Ridley, N., & Gulish, A. (2018). Three educational pathways to good jobs: High school, middle skills, and bachelor’s degree. https://1gyhoq479ufd3yna29x7ubjn-wpengine.netdna-ssl.com/wp-content/uploads/3ways-FR.pdf
8. Coleman, J. S., Hoffer, T., & Kilgore, S. (1982). Cognitive outcomes in public and private schools. Sociology of Education, 55(2-3), 65-76. https://www.jstor.org/stable/pdf/2112288.pdf?casa_token=2urqntYyuZQAAAAA:Rj2XcpFpD4Asklsmj_minXEoi7CsxMD1kg7yrb81rVd2wuN_j7zMZu6feBoRaNnN53xFobtwRojV4LkD8cv8WzWUgMWGGMbIBgaVwN2cMWCRp8G0VoI

9. De Boeck, P., & Wilson, M. (Eds.) (2004). Explanatory item response models. Springer.
10. Eriksson, K., Björnstjerna, M., & Vartanova, I. (2020). The relation between gender egalitarian values and gender differences in academic achievement. Frontiers in Psychology, 11. https://doi.org/10.3389/fpsyg.2020.00236
11. Figlio, D.N., & Stone, J.A. (1997). School choice and student performance: Are private schools really better? (Discussion Paper 1141-97). Institute for Research on Poverty, University of Wisconsin-Madison.
12. Gierl, M., Khaliq, S. N., & Boughton, K. (1999, June). Gender differential item functioning in mathematics and science: Prevalence and policy implications. In Annual Meeting of the Canadian Society for the Study of Education, Canada.
13. Hambleton, R., Swaminathan, H., & Rogers, H. (1991). Fundamentals of item response theory. Sage.
14. Hughes, G. (2001). Exploring the availability of student scientist identities within curriculum discourse: An anti-essentialist approach to gender-inclusive science. Gender & Education, 13(3), 275-290. https://doi.org/10.1080/09540250120063562
15. Hyde, J. S. (2005). The gender similarities hypothesis. American Psychologist, 60(6), 581-592. https://doi.org/10.1037/0003-066X.60.6.581
16. Hyde, J. S., & Linn, M. C. (2006). Gender similarities in mathematics and science. Science, 314(5799), 599-600. https://doi.org/10.1126/science.1132154
17. International Association for the Evaluation of Educational Achievement. (1988). Science achievement in seventeen countries (A Preliminary Report). Pergamon Press.
18. Kalaycioğlu, D. B., & Berberoğlu, G. (2010). Differential item functioning analysis of the science and mathematics items in the university entrance examinations in Turkey. Journal of Psychoeducational Assessment, 29(5), 467-478. https://doi.org/10.1177/0734282910391623
19. Kan, A., Bulut, O., & Cormier, D. C. (2018). The impact of item stem format on the dimensional structure of mathematics assessments. Educational Assessment, 24(1), 13-32. https://doi.org/10.1080/10627197.2018.1545569
20. Lee, V. E., & Burkam, D. T. (1996). Gender differences in middle grade science achievement: Subject domain, ability level, and course emphasis. Science Education, 80(6), 613-650. https://doi.org/10.1002/(SICI)1098-237X(199611)80:6<613::AID-SCE1>3.0.CO;2-M
21. Legewie, J., & DiPrete, T. A. (2014). The high school environment and the gender gap in science and engineering. Sociology of Education, 87(4), 259-280. https://doi.org/10.1177/0038040714547770
22. Liou, P. Y., & Bulut, O. (2020). The effects of item format and cognitive domain on students’ science performance in TIMSS 2011. Research in Science Education, 50(1), 99-121. https://doi.org/10.1007/s11165-017-9682-7
23. Liu, O. L., & Wilson, M. (2009). Gender differences in large-scale math assessments: PISA trend 2000 and 2003. Applied Measurement in Education, 22(2), 164-184. https://doi.org/10.1080/08957340902754635
24. Martinková P., & Drabinová A. (2018) ShinyItemAnalysis for teaching psychometrics and to enforce routine analysis of educational tests. The R Journal, 10(2), 503-515. https://doi.org/10.32614/RJ-2018-074
25. Meinck, S., & Brese, F. (2019). Trends in gender gaps: Using 20 years of evidence from TIMSS. Large-Scale Assessments in Education, 7(1), 1-23. https://doi.org/10.1186/s40536-019-0076-3
26. Ministry of National Education. (2015). Ortaöğretim kurumlarına geçiş uygulaması tercih ve yerleştirme e-kılavuzu 2015. http://odsgm.meb.gov.tr/meb_iys_dosyalar/2015_05/28024630_ekilavuz28.05.2015.pdf
27. Mullis, I. V. S., Martin, M. O., Foy, P., Kelly, D. L., & Fishbein, B. (2020). TIMSS 2019 international results in mathematics and science. https://timssandpirls.bc.edu/timss2019/international-results/
28. Newhouse, D., & Beegle, K. (2006). The effect of school type on academic achievement evidence from indonesia. Journal of Human Resources, 41(3), 529-557. https://doi.org/10.3368/jhr.XLI.3.529
29. Ober, C., Loisel, D. A., & Gilad, Y. (2008). Sex-specific genetic architecture of human disease. Nature Reviews Genetics, 9(12), 911-922. https://doi.org/10.1038/nrg2415
30. Organisation for Economic Co-operation and Development. (2016). PISA 2015 results: Excellence and equity in education (Vol. 1). OECD Publishing. https://doi.org/10.1787/9789264266490-en
31. Organisation for Economic Co-operation and Development. (2021). Education at a Glance 2021: OECD indicators. OECD Publishing. https://doi.org/10.1787/b35a14e5-en
32. Petscher, Y., Compton, D. L., Steacy, L., & Kinnon, H. (2020). Past perspectives and new opportunities for the explanatory item response model. Annals of Dyslexia, 70(2), 160-179. https://doi.org/10.1007/s11881-020-00204-y
33. Quinn, D. M., & Cooc, N. (2015). Science achievement gaps by gender and race/ethnicity in elementary and middle school. Educational Researcher, 44(6), 336-346. https://doi.org/10.3102/0013189X15598539
34. R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing.
35. Reilly, D., Neumann, D. L., & Andrews, G. (2015). Sex differences in mathematics and science achievement: A meta-analysis of National Assessment of Educational Progress Assessments. Journal of Educational Psychology, 107(3), 645-662. https://doi.org/10.1037/edu0000012
36. Rijmen, F., Tuerlinckx, F., De Boeck, P., & Kuppens, P. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8(2), 185-205. https://doi.org/10.1037/1082-989X.8.2.185
37. Sinnes, A. T., & Løken, M. (2014). Gendered education in a gendered world: looking beyond cosmetic solutions to the gender gap in science. Cultural Studies of Science Education, 9(2), 343-364. https://doi.org/10.1007/s11422-012-9433-z
38. Stoet, G., & Geary, D. C. (2018). The gender-equality paradox in science, technology, engineering, and mathematics education. Psychological Science, 29(4), 581-593. https://doi.org/10.1177/0956797617741719
39. Wang, M.-T., Eccles, J. S., & Kenny, S. (2013). Not lack of ability but more choice: Individual and gender differences in choice of careers in science, technology, engineering, and mathematics. Psychological Science, 24(5), 770-775. https://doi.org/10.1177/0956797612458937
40. Wilson, M., De Boeck, P., & Carstensen, C. H. (2008). Explanatory item response models: A brief introduction. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 91-120). Hogrefe.
41. Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125-145. https://doi.org/10.1177/014662168400800201
42. Young, D. J., & Fraser, B. J. (1994). Gender differences in science achievement: Do school effects make a difference? Journal of Research in Science Teaching, 31(8), 857-871. https://doi.org/10.1002/tea.3660310808
43. Young, D. J., & Fraser, B. J. (1992, April). Sex differences in science achievement: A multilevel analysis. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco. https://files.eric.ed.gov/fulltext/ED356947.pdf
44. Zhang, D., & Campbell, T. (2015). An examination of the impact of teacher quality and “opportunity gap” on student science achievement in China. International Journal of Science and Mathematics Education, 13(3), 489-513. https://doi.org/10.1007/s10763-013-9491-z