The Impact of Missing Data on the Performances of DIF Detection Methods

Abstract

This study analyzed the impact of missing data techniques on performances of two differential item functioning (DIF) detection methods (Mantel Haenszel and Multiple Indicator and Multiple Causes) under missing completely at random missing data mechanism. Percentage of missing data was set at 5% and 15%. Zero imputation, listwise deletion and fractional hot-deck imputation were used to handle missing data. The data set of the study consisted of 17 items in the S12 item cluster of Programme for International Student Assessment (PISA) 2015 science test. Results showed that fractional hot-deck imputation produced the best results in identifying DIF items in all conditions and it had also the closest DIF values to the values obtained from complete data set. It was also found that multiple indicator and multiple causes method was more adversely affected than Mantel Haenszel by the presence of missing data.

Keywords

• Differential item functioning
• Mantel Haenszel
• MIMIC
• missing data

References

1. Banks, K. (2015). An introduction to missing data in the context of differential item functioning. Practical Assessment, Research & Evaluation, 20(12), 1-10. https://eric.ed.gov/?id=EJ1059748
2. Banks, K., & Walker, C. (2006, April). Performance of SIBTEST when focal group examinees have missing data. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
3. Camilli, G., & Shepard, L. A. (1994). Methods for identifying biased test items. London Sage.
4. Clauser, B. E., & Mazor, K. M. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement Issues and Practice, 17(1), 31-44. https://eric.ed.gov/?id=EJ564712
5. Doğanay Erdoğan, B. (2012). Çoklu atama yöntemlerinin Rasch modelleri için performansının benzetim çalışması ile incelenmesi [Assessing the performance of multiple imputation techniques for Rasch models with a simulation study] (Publication No. 314412) [Doctoral dissertation, Ankara University]. Council of Higher Education Thesis Center.
6. Dong, Y., & Peng, C. Y. (2013). Principled missing data methods for researchers. Springer Plus, 2(1), 222. https://doi.org/10.1186/2193-1801-2-222
7. Emenogu, B. C., Falenchuk, O., & Childs, R. A. (2010). The effect of missing data treatment on Mantel-Haenszel DIF detection. The Alberta Journal of Educational Research, 56(4), 459-469. https://doi.org/10.11575/ajer.v56i4.55429
8. Finch, H. (2011a). The use of multiple imputation for missing data in uniform DIF analysis: Power and type I error rates. Applied Measurement in Education, 24(4), 281-301. https://doi.org/10.1080/08957347.2011.607054

9. Finch, H. (2011b). The impact of missing data on the detection of nonuniform differential item functioning. Educational and Psychological Measurement, 71(4), 663-683. https://doi.org/10.1177/0013164410385226
10. Finch, H. (2005). The MIMIC model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Applied Psychological Measurement, 29(4), 278-295. https://doi.org/10.1177/0146621605275728
11. Finch, H. W., & French, B. F. (2007). Detection of crossing differential item functioning a comparison of four methods. Educational and Psychological Measurement, 67(4), 565-582. https://doi.org/10.1177/0013164406296975
12. Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2012). How to design and evaluate research in education McGraw-hill.
13. Garrett, P. L. (2009). A monte carlo study investigating missing data, differential item functioning, and effect size (Publication No. 3401601) [Doctoral dissertation, Georgia State University]. ProQuest Dissertations Publishing.
14. Hallquist, M., & Wiley, J. (2018). MplusAutomation: An R package for facilitating large-scale latent variable analyses in Mplus. Structural Equation Modeling: A Multidisciplinary Journal, 25(4), 621-638. https://doi.org/10.1080/10705511.2017.1402334
15. Harrington, D. (2009). Confirmatory factor analysis. Oxford University Press.
16. Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer, & H. I. Braun, Test Validity (pp. 129-145). Lawrence Erlbaum.
17. Holland, P. W., & Wainer, H. (1993). Differential item functioning. Lawrence Erlbaum.
18. Im, J., Cho, I. H., & Kim, J. K. (2018). FHDI: Fractional hot deck and fully efficient fractional imputation. https://CRAN.R-project.org/package=FHDI
19. Im, J., Kim, J. K., & Fuller, W. A. (2015). Two-phase sampling approach to fractional hot deck imputation. In Proceedings of the Survey Research Methods Section, pages 1030-1043. http://www.asasrms.org/Proceedings/y2015/files/233957.pdf
20. Jin, KY., & Chen, HF. (2020). MIMIC approach to assessing differential item functioning with control of extreme response style. Behavior Research Methods, 52, 23-35. https://doi.org/10.3758/s13428-019-01198-1
21. Kalton, G., & Kish, L. (1984). Some efficient random imputation methods. Communications in Statistics-Theory and Methods ,13(16), 1919-1939. https://doi.org/10.1080/03610928408828805
22. Kim, J. K., & Fuller, W. (2004). Fractional hot deck imputation. Biometrika, 91(3), 559-578. https://doi.org/10.1093/biomet/91.3.559
23. Magis, D., Beland, S., Tuerlinckx, F., & De Boeck, P. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847-862. https://doi.org/10.3758/BRM.42.3.847
24. Montoya, A. K., & Jeon, M. (2020). MIMIC models for uniform and nonuniform DIF as moderated mediation models. Applied Psychological Measurement, 44(2), 118-136. https://doi.org/10.1177/0146621619835496
25. Peugh, J. L., & Enders, C. K. (2004). Missing data in educational research:A review of reporting practices and suggestions for improvement. Review of Educational Research, 74(4), 525-556. https://journals.sagepub.com/doi/pdf/10.3102/00346543074004525
26. Robitzsch, A., & Rupp, A. A. (2009). Impact of missing data on the detection of differential item functioning: The case of Mantel-Haenszel and logistic regression analysis. Educational and Psychological Measurement, 69(1), 18-34. https://doi.org/10.1177/0013164408318756
27. Rousseau, M., Bertrand, R., & Boiteau, N. (2004, April). Impact of missing data on robustness of DIF IRT-based and non IRT-based methods. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
28. Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581-592. https://doi.org/10.1093/biomet/63.3.581
29. Scheuneman, J. (1979). A method of assessing bias in test items. Journal of Educational Measurement, 16, 143–152. https://www.jstor.org/stable/1433816
30. Shih, C. L., & Wang, W. C. (2009). Differential item functioning detection using multiple indicators, multiple causes method with a pure short anchor. Applied Psychological Measurement, 33(3), 184-199. https://doi.org/10.1177/0146621608321758
31. Tamcı, P. (2018). Kayıp veriyle baş etme yöntemlerinin değişen madde fonksiyonu üzerindeki etkisinin incelenmesi [Investigation of the impact of techniques of handling missing data on differential item functioning] (Publication No. 517260) [Master's dissertation, Hacettepe University]. Council of Higher Education Thesis Center.
32. Uğurlu, S., & Atar, B. (2020). Performances of MIMIC and logistic regression procedures in detecting DIF. Journal of Measurement and Evaluation in Education and Psychology, 11(1), 1-12. https://doi.org/10.21031/epod.531509
33. Woods, C. M. (2009). Evaluation of MIMIC-model methods for DIF testing with comparison to two-group analysis. Multivariate Behavioral Research, 44(1), 1-27. https://doi.org/10.1080/00273170802620121
34. Zieky, M. (1993). Practical questions in the use of DIF statistics in test development. In P. W. Holland, & H. Wainer, Differential Item Functioning (pp. 337-347). Lawrence Erlbaum.
35. Zumbo, B. D. (2007). Three generation of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223–233. https://doi.org/10.1080/15434300701375832