An Investigation of the Effect of Missing Data on Differential Item Functioning in Mixed Type Tests

Abstract

In this research, the aim was to examine the effects of Markov Chain Monte Carlo (MCMC), multiple imputation (MI), and expectation maximization (EM), all methods of coping with missing data in mixed type tests including dichotomous and polytomous items, on the differential item functioning (DIF). The study was carried out on a complete data set consisting of the scores of 1160 students who took booklet number 9 in the science test in Trends in International Mathematics and Science Study (TIMSS) 2019 and answered it in full. The conditions to be examined for the effectiveness of the methods were missing data mechanism (MCAR and MAR), DIF level (A, B, and C), and missing data rate (10% and 20%). Data were assigned to the missing data sets created by deleting data at different rates under the missing completely at random (MCAR) and missing at random (MAR) mechanisms over the aforementioned data set. DIF analysis was performed on all the data sets obtained with the poly-SIBTEST method using the MCMC, MI, and EM methods. The results obtained from the complete data set were then compared with the result implications from other data sets for reference. The study showed that the EM and MCMC methods performed better for the C-level DIF than the A and B levels in terms of all conditions examined. MI was observed to be the most successful method in determining DIF in items showing DIF in 10% and 20% MCAR mechanisms. When compared with the complete data set, the three methods showed similar results in the 10% MAR mechanism while MCMC gave the closest results in the 20% MAR mechanism.

Keywords

• mixed type test
• missing data
• differential item functioning
• poly-SIBTEST

References

1. Allison, P. D. (2002). Missing data. Sage.
2. Alpar, R. (2021). Çok değişkenli istatistiksel yöntemler. Detay.
3. Banks, K. (2015). An introduction to missing data in the context of differential item functioning. Practical Assessment, Research & Evaluation, 20(12), 12. https://doi.org/10.7275/FPG0-5079
4. Banks, K., & Walker, C. M. (2006). Performance of SIBTEST when focal group examinees have missing data. Paper presented at the annual meeting of the National Council on Measurement in Education.
5. Bolt, D. M. (2000). A SIBTEST approach to testing DIF hypotheses using experimentally designed test items. Journal of Educational Measurement, 37(4), 307-327. https://doi.org/10.1111/j.1745-3984.2000.tb01089.x
6. Camilli, G. (2006). Test fairness. In R. L. Brennan (Ed.), Educational measurement (4th ed.). American Council on Education & Praeger Publishers.
7. Çüm, S., Demir, E. K., Gelbal, S., & Kışla, T. (2018). A comparison of advanced methods used for missing data imputation under different conditions. Mehmet Akif Ersoy University Journal of Education Faculty, 45, 230-249. https://doi.org/10.21764/maeuefd.332605
8. Demir, E. (2013). Item and test parameters estimations for multiple choice tests in the presence of missing data: The case of SBS. Journal of Educational Sciences Research, 3(2), 47-68. http://dx.doi.org/10.12973/jesr.2013.324a

9. Emenogu, B. C., Falenchuck, O., & Childs, R. A. (2010). The effect of missing data treatment on Mantel- Haenszel DIF detection. The Alberta Journal of Educational Research, 56(4), 459-469. https://doi.org/10.11575/ajer.v56i4.55429
10. Enders, C. K. (2010). Applied missing data analysis. The Guilford Press.
11. Falenchuk, O., & Herbert, M. (2009). Investigation of differential non-response as a factor affecting the results of Mantel-Haenszel DIF detection. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA.
12. Fang, T. (1999). Detecting DIF in polytomous item responses [Doctoral dissertation, University of Ottawa]. https://ruor.uottawa.ca/handle/10393/8495?locale=fr
13. Finch, H. (2011a). The use of multiple imputation for missing data in uniform DIF analysis: Power and type I error rates. Applied Measurement in Education, 24, 281-301. https://doi.org/10.1080/08957347.2011.607054
14. Finch, H. W. (2011b). The impact of missing data on the detection of nonuniform differential item functioning. Educational and Psychological Measurement, 71(4), 663-683. https://doi.org/10.1177/0013164410385226
15. Garrett, P. (2009). A Monte Carlo study investigating missing data, differential item functioning and effect size [Doctoral thesis, Georgia State University]. https://doi.org/10.57709/1060078
16. Gierl, M. J. (2005). Using dimensionality-based DIF analysis to identify and interpret constructs that elicit group differences. Educational Measurement: Issues and Practice, 24, 3-14. https://doi.org/10.1111/j.1745-3992.2005. 00002.x
17. Gill, J. (2002). Bayesian methods: A social and behavioral sciences approach. Chapman and Hall.
18. Gök, B., Kelecioğlu, H., & Dogan, N. (2010). The comparison of Mantel-Haenszel and logistic regression techniques in determining the differential item functioning. Education and Science, 35(156), 3-16.
19. Hambleton, R. K., Clauser, B. E., Mazor, K. M. & Jones, R. W. (1993). Advances in the detection of differentially functioning test items. University of Massachusetts, School of Education. http://files.eric.ed.gov/fulltext/ED356264.pdf
20. Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the DIF analysis. Test Validity, 129-145.
21. Karasar, N. (2011). Bilimsel araştırma yöntemleri. Nobel.
22. Little, R., & Rubin, D. (2020). Statistical analysis with missing data (4th ed.). Wiley.
23. Millsap, R. E., & Everson, H. T. (1993). Methodology review: Statistical approaches for assessing measurement bias. Applied Psychological Measurement, 17, 297-334. https://doi.org/10.1177/014662169301700401
24. Nichols, E., Deal, J. A., Swenor, B. K., Abraham, A. G., Armstrong, N. M., Bandeen-Roche, K., Carlson, M.C., Grisworld, M., Lin, F. R., Mosley, T. H., Ramulu, P. Y., Reed, N. S., Sharrett, A. R., & Gross, A. L. (2022). The effect of missing data and imputation on the detection of bias in cognitive testing using differential item functioning methods. BMC Medical Research Methodology, 22(1), 1-12. https://doi.org/10.1186/s12874-022-01572-2
25. Özgüven, İ. E. (2017). Psikolojik testler. Nobel.
26. Robitzsch, A., & Rupp, A. A. (2009). Impact of missing data on the detection of differential item functioning: The case of Mantel-Haenszel and Logistic Regression analysis. Educational and Psychological Measurement, 69(1), 18-34. https://doi.org/10.1177/0013164408318756
27. Roussos, L. A., & Stout, W. F. (1996). Simulation studies of the effects of small sample size and studied item parameters on SIBTEST and Mantel‐Haenszel type I error Performance. Journal of Educational Measurement, 33(2), 215-230. https://doi.org/10.1111/j.1745-3984.1996.tb00490.x
28. Rousseau, M., Bertrand, R., & Boiteau, N. (2006). Impact of missing data treatment on the efficiency of DIF methods. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
29. Rubin D. B. (1976). Inference and missing data. Biometrika, 72, 359-364.
30. Selvi, H., & Alıcı, D. (2018). Investigating the impact of missing data handling methods on the detection of differential item functioning. International Journal of Assessment Tools in Education, 5(1), 1-14. https://doi.org/10.21449/ijate.330885
31. Sedivy, S. K., Zhang, B., & Traxel, N. M. (2006). Detection of differential item functioning with polytomous items in the presence of missing data. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
32. Schafer, J. L., & Olsen, M. K. (1998). Multiple imputation for multivariate missing-data problems: A data analyst's perspective. Multivariate Behavioral Research, 33(4), 545-571.
33. Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147-177. https://doi.org/10.1037/1082-989X.7.2.147
34. Shealy, R., & Stout, W. (1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58(2), 159-194. https://doi.org/10.1007/BF02294572
35. Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361-370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x
36. Tabachnick, B., & Fidell, L. (1996). Using multivariate statistics (3rd ed.). Herper Collins College Publishers. Tamcı, P. (2018). Kayıp veriyle başa çıkma yöntemlerinin değişen madde fonksiyonu üzerindeki etkisinin incelenmesi [Investigation of the impact of techniques of handling missing data on differential item functioning] (Thesis No. 517260) [Master’s thesis, Hacettepe University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
37. Toka, O. (2012). Kayıp veri durumunda sağlam kestirim [Robust estimation in case of missing data] (Thesis No. 321449) [Master’s thesis, Hacettepe University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
38. Turgut, M. F., & Baykul, Y. (2012). Eğitimde ölçme ve değerlendirme []. Pegem.
39. Van Buuren, S. (2012). Flebitler imputation of missing data. CRC Press.
40. Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Directorate of Human Resources Research and Evaluation, Department of National Defense.