Investigating Differential Item Functioning of an International Mathematics Competition Items across Gender Groups

Öz

Mathematical problem-solving competitions have existed for over a century. Scholars report the gender gap in these competitions. As a result, it is necessary to determine whether any score difference between gender groups is attributable to a genuine difference or is the result of the exam itself. Thus, the current study specifically examined bias in one of the well-known mathematics competitions: the Kangaroo Mathematics competition. Determining the fairness of Kangaroo mathematics competition items across gender groups is crucial for creating accurate comparisons and avoiding unintended construct irrelevant bias. To examine the bias, Differential Item Functioning (DIF) analyses were conducted using Logistic Regression, Mantel-Haenszel, and Item Response Theory Likelihood Ratio Test DIF detection methods. After a series of investigations, out of 336 items, it was concluded that these mathematics items were free of DIF and bias across the gender groups. Further implications were discussed in detail regarding the validity and bias.

Anahtar Kelimeler

• DIF
• bias
• mathematical problem-solving competitions
• logistic regression
• Mantel-Haenszel
• Item Response Theory Likelihood Ratio Test

Uluslararası Matematik Yarışması Maddelerinin Cinsiyet Gruplarına Göre Değişen Madde Fonksiyonu (DMF) açısından İncelenmesi

Öz

Matematiksel problem çözme yarışmaları bir asırdan fazla bir süredir uygulanmaktadır. Araştırmalar bu yarışmalarda kızlar ve erkekler arasında başarı farkı olduğunu göstermektedir. Cinsiyet grupları arasındaki puan farkının gerçekte var olan bir farklılıktan mı yoksa sınavdaki sorulardan mı kaynaklandığının belirlenmesi değerli bilgiler sunacaktır. Bu nedenle bu çalışma pek çok öğrencinin katıldığı matematik yarışmalarından biri olan Uluslararası Kanguru Matematik yarışmasındaki madde yanlılığını incelemektedir. Maddelerin Değişen Madde Fonksiyonu (DMF) içerme durumlarını incelemek için Lojistik Regresyon, Mantel-Haenszel ve Madde Tepki Kuramı Olabilirlik Oranı Testi kullanılmıştır. Analizlerden elde edilen bulgulara göre toplam 336 maddeden oluşan bu matematik testlerinin DMF içermediği, dolayısıyla maddelerin cinsiyet grupları arasında yanlılık yaratmadığı sonucuna varılmıştır. Makalede geçerlilik ve yanlılıkla ilgili diğer çıkarımlar ayrıntılı olarak tartışılmıştır.

Anahtar Kelimeler

• DMF
• yanlılık
• uluslararası matematik yarışmaları
• lojistik regresyon
• Mantel-Haenszel ve Madde Tepki Kuramı Olabilirlik Oranı Testi

Kaynakça

1. Akveld, M., Caceres-Duque, L. F., & Geretschläger, R. (2020). Math Kangaroo. Mathematics Competitions, 33(2), 48–66.
2. Akveld, M., Caceres-Duque, L. F., Nieto Said, J. H., & Sánchez Lamoneda, R. (2020). The Math Kangaroo Competition. Espacio Matemático, 1(2), 74–91.
3. Allalouf, A., Hambleton, R. K., & Sireci, S. G. (1999). Identifying the causes of DIF in translated verbal items. Journal of Educational Measurement, 36(3), 185–198. https://doi.org/10.1111/j.1745-3984.1999.tb00553.x
4. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for educational and psychological testing. Washington, D.C: American Educational Research Association.
5. Andritsch, L., Hauke, E., & Kelz, J. (2020). How to create and solve: Analysis of items from the Mathematical Kangaroo from two perspectives. In R. Geretschläger (Ed.), Engaging young students in mathematics through competitions—World perspectives and practices, Vol. II: Mathematics competitions and how they relate to research, teaching and motivation (pp. 117–136). World Scientific.
6. Applebaum, M., & Leikin, R. (2019). Girls’ performance in the Kangaroo contest. In M. Nolte (Ed.), Including the Highly Gifted and Creative Students–Current Ideas and Future Directions-Proceedings of the 11th International Conference on Mathematical Creativity and Giftedness (mcg 11), (pp. 87–94). Hamburg, Germany.
7. Arikan, S. (2019). Are Differentially Functioning Mathematics Items Reason of Low Achievement of Turkish Students in PISA 2015? Journal of Measurement and Evaluation in Education and Psychology, 10(1), 49–67. https://doi:10.21031/epod.466860
8. Baniasadi, A., Salehi, K., Khodaie, E., Bagheri Noaparast, K., & Izanloo, B. (2023). Fairness in classroom assessment: A systematic review. The Asia-Pacific Education Researcher, 32, 91–109. https://doi.org/10.1007/s40299-021-00636-z

9. Berrío, Á. I., Gomez-Benito, J., & Arias-Patiño, E. M. (2020). Developments and trends in research on methods of detecting differential item functioning. Educational Research Review, 31, 100340. https://doi.org/10.1016/j.edurev.2020.100340
10. Cohen, J. (1988). Statistical power analysis for the behavioral sciences, (2nd ed.). Hillsdale, NJ: Erlbaum.
11. de Losada, M. F., & Taylor, P. J. (2022). Perspectives on mathematics competitions and their relationship with mathematics education. ZDM–Mathematics Education, 54(5), 941–959. https://doi.org/10.1007/s11858-022-01404-z
12. Desjarlais, M. A. (2009). Gender differences on the American Mathematics Competition AMC 8 contest. The University of Nebraska-Lincoln.
13. Donner, L., Kelz, J., Stipsits, E., & Stuhlpfarrer, D. (2021). Which test-wiseness based strategies are used by Austrian winners of the Mathematical Kangaroo?. Mathematics Competitions, 34(1), 88–101.
14. Dorans, N. J. (2013). ETS contributions to the quantitative assessment of item, test, and score fairness. ETS Research Report Series, 2013(2), i–38.
15. Escardibul, J. O., & Mora, T. (2013). Teacher gender and student performance in Mathematics. Evidence from Catalonia (Spain). Journal of Education and Training Studies, 1(1), 39–46.
16. ETS. (2022). ETS guidelines for developing fair tests and communications. https://www.ets.org/content/dam/ets-org/pdfs/about/fair-tests-and-communications.pdf
17. Field, A. (2013). Discovering statistics using IBM SPSS statistics. London: Sage.
18. Geretschläger, R., & Donner, L. (2022). Writing and choosing problems for a popular high school mathematics competition. ZDM–Mathematics Education, 54(5), 971–982. https://doi.org/10.1007/s11858-022-01351-9
19. Gneezy, U., Muriel, N., & Aldo, R. (2003). Performance in competitive environments: Gender differences. Quarterly Journal of Economics, 118(3), 1049–1074. https://doi.org/10.1162/00335530360698496
20. He, J., & van de Vijver, F. J. R. (2013). Methodological issues in cross-cultural studies in educational psychology. In G. A. D. Liem & A. B. I. Bernardo (Eds.), Advancing cross-cultural perspectives on educational psychology: A festschrift for Dennis McInerney (pp. 39–56). Charlotte, NC: Information Age Publishing.
21. Holland, P. W., & Thayer, D. T. (1986). Differential item functioning and the Mantel-Haenszel procedure (ETS Research Report No. RR-86-31). Princeton, NJ: ETS.
22. Holland, P. W., & Thayer, D. T. (1988). Differential item performance and Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity, (pp.129–145). Hillsdale, N.J.: Erlbaum.
23. Hyde, J. S., Lindberg, S. M., Linn, M. C., Ellis, A. B., & Williams, C. C. (2008). Gender similarities characterize math performance. Science, 321(5888), 494–495. https://doi.org/10.1126/science.1160364
24. International Test Commission. (2001). International guidelines for test use. International Journal of Testing, 1(2), 93–114. https://doi.org/10.1207/S15327574IJT0102_1
25. Jiang, P., & Xiong, B. (2021, April). Analyze the quality of Math Kangaroo problems with a content analysis. In Journal of Physics: Conference Series (Vol. 1875, No. 1, p. 012015). IOP Publishing.
26. Jodoin, M. G., & Gierl, M. J. (2001). Evaluating type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329–349. https://doi.org/10.1207/S15324818AME1404_2
27. Kankaraš, M., & Moors, G. (2014). Analysis of cross-cultural comparability of PISA 2009 scores. Journal of Cross-Cultural Psychology, 45(3), 381–399. https://doi.org/10.1177/0022022113511297
28. Lyons-Thomas, J., Sandilands, D. D., & Ercikan, K. (2014). Gender differential item functioning in Mathematics in four international jurisdictions. Education &Science, 39(172), 20–32.
29. Mellroth, E. (2015). Problem solving competency and the mathematical kangaroo. In K. Krainer & N. Vondrová (Eds.), Proceedings of the Ninth Congress of the European Society for Research in Mathematics Education (CERME9, 4-8 February 2015) (pp. 1095–1096). Prague, Czech Republic: Charles University in Prague, Faculty of Education and ERME. https://hal.science/CERME9/public/CERME9_Proceedings_2015.pdf
30. Messick, S. (1989). Validity. In R. Linn (Ed.), Educational measurement (3rd edition, pp.13–103). New York, NY: Macmillan.
31. Niederle, M., & Vesterlund, L. (2010). Explaining the gender gap in Math test scores: The Role of Competition. Journal of Economic Perspectives, 24(2), 129–144. https://doi.org/10.1257/jep.24.2.129
32. Penfield, R. D. (2005). DIFAS: Differential Item Functioning Analysis System. Applied Psychological Measurement, 29, 150–151. https://doi.org/10.1177/0146621603260686
33. Reynolds, K., Khorramdel, L., & von Davier, M. (2022). Can students’ attitudes towards mathematics and science be compared across countries? Evidence from measurement invariance modeling in TIMSS 2019. Studies in Educational Evaluation, 74, 101169. https://doi.org/10.1016/j.stueduc.2022.101169
34. Roberson, N. D., & Zumbo, B. D. (2019). Migration background in PISA’s measure of social belonging: Using a diffractive lens to interpret multi-method DIF studies. International Journal of Testing, 19(4), 363–389. https://doi.org/10.1080/15305058.2019.1632316
35. Roth, W.-M., Oliveri, M. E., Sandilands, D., Lyons-Thomas, J., & Ercikan, K. (2013). Investigating sources of differential item functioning using expert think-aloud protocols. International Journal of Science Education, 35, 546–576. https://doi.org/10.1080/09500693.2012.721572
36. Rubright, J. D., Jodoin, M., Woodward, S., & Barone, M. A. (2022). Differential item functioning analysis of United States medical licensing examination step 1 items. Academic Medicine, 97(5), 718–722. https://doi.org/10.1097/ACM.0000000000004567
37. Slocum-Gori, S. L., & Zumbo, B. D. (2011). Assessing the unidimensionality of psychological scales: Using multiple criteria from factor analysis. Social Indicators Research, 102, 443–461. https://doi.org/10.1007/s11205-010-9682-8
38. Stark, S., Chernyshenko, O. S., & Drasgow, F. (2004). Examining the effects of differential item (functioning and differential) test functioning on selection decisions: When are statistically significant effects practically important?. Journal of Applied Psychology, 89(3), 497–508. https://doi.org/10.1037/0021-9010.89.3.497
39. Steinberg, L., & Thissen, D. (2006). Using effect sizes for research reporting: examples using item response theory to analyze differential item functioning. Psychological methods, 11(4), 402–415. https://doi.org/10.1037/1082-989X.11.4.402
40. Thissen, D. (2001). IRTLRDIF v2.0b: Software for the computation of the statistics involved in item response theory likelihood-ratio tests for differential item functioning [Documentation for computer program]. L.L. Thurstone Psychometric Laboratory, University of North Carolina at Chapel Hill.
41. Thissen, D., Steinberg, L., & Wainer, H. (1993). Detection of differential item functioning using the parameters of item response models. In P. W. Holland & H. Wainer (Eds.), Differential Item Functioning, (pp.67–113). Mahwah, NJ: Erlbaum.
42. van de Vijver, F. J. R., & Leung, K. (1997). Methods and data analysis of comparative research. Thousand Oaks, CA: Sage.
43. van de Vijver, F., & Tanzer, N. K. (2004). Bias and equivalence in cross-cultural assessment: An overview. Revue Européenne de Psychologie Appliquée/European Review of Applied Psychology, 54(2), 119–135. https://doi.org/10.1016/j.erap.2003.12.004
44. Wedman, J. (2018). Reasons for gender-related differential item functioning in a college admissions test. Scandinavian Journal of Educational Research, 62(6), 959–970. https://doi.org/10.1080/00313831.2017.1402365
45. Zieky, M. (1993). Practical questions in the use of DIF statistics in item development. In P. W. Holland & H. Wainer (Eds.), Differential Item Functioning, (337–364). Hillsdale, NJ: Lawrence Erlbaum.
46. Zumbo, B. D. (1999). A Handbook on the Theory and Methods of Differential Item Functioning (DIF): Logistic Regression Modeling as a Unitary Framework for Binary and Likert-Type (Ordinal) Item Scores. Ottawa, ON: Directorate of Human Resources Research and Evaluation, Department of National Defense.
47. Zumbo, B. D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223–233. https://doi.org/10.1080/15434300701375832
48. Zumbo, B. D., & Gelin, M. N. (2005). A matter of test bias in educational policy research: Bringing the context into picture by investigating sociological/community moderated (or mediated) test and item bias. Journal of Educational Research & Policy Studies, 5(1), 1–23.
49. Zumbo, B. D., & Thomas, D. R. (1997). A measure of effect size for a model-based approach for studying DIF. Prince George, Canada: Edgeworth Laboratory for Quantitative Behavioral Science, University of Northern British Columbia.
50. Zwick, R., & Ercikan, K. (1989). Analysis of differential item functioning in the NAEP History assessment. Journal of Educational Measurement, 26, 55–66. https://doi.org/10.1111/j.1745-3984.1989.tb00318.x
51. Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99(3), 432–442. https://doi.org/10.1037/0033-2909.99.3.432