Investigating the Impact of Missing Data Handling Methods on the Detection of Differential Item Functioning

Abstract

In this study, it is aimed to investigate the impact of different missing data handling methods on the detection of Differential Item Functioning methods (Mantel Haenszel and Standardization methods based on Classical Test Theory and Likelihood Ratio Test method based on Item Response Theory). In this regard, on the data acquired from 1046 candidates who entered to Foreign National Student Exam (FNSE) held in year 2016 by Mersin University (MEU) and answered Basic Skills subtest, using different missing data handling methods, differential item functioning analyses with Mantel Haenszel, Standardization and Likelihood Ratio Test methods are performed. Basic Skills test consists of 80 multiple choice items. The items are all binary scored (1-0) items. Among the participants 523 are female and 523 are male. The findings showed that the number of items flagged as DIF has changed with the used missing data handling methods. The DIF detection methods based on Classical Test Theory are more consistent within themselves compared to DIF detection method based on Item Response Theory, whereas the used missing data handling methods differentiate the DIF detected items and this difference reaches a significant level for Mantel Haenszel method

Keywords

• Differential Item Functioning DIF,Test and Item Bias,Missing Values; Imputation of Missing Data,Mantel Haenszel,Likelihood Ratio Test

References

1. Abedlazeez, N. (2010). Exploring DIF: Comparison of CTT and IRT methods. International Journal of Sustainable Development, 7(1), 11-46.
2. Allison, P. D. (2002). Missing data. California: Sage Publication Inc.
3. Alpar, R. (2011). Uygulamalı çok değişkenli istatistiksel yöntemler. Ankara: Detay Yayıncılık.
4. Angoff, W.H. (1993). Perspectives on differential item functioning methodology. In Holland & Wainer (Ed.), Differential Item Functioning. New Jersey: Lawrence Erlbaum Associates Publishers.
5. Banks, K., & Walker, C. (2006). Performance of SIBTEST when focal group examinees have missing data. San Francisco: National Council of Measurement in Education.
6. Banks, K. (2015). An introduction to missing data in the context of differential item functioning. Practical Assessment, Research & Evaluation. 20(12).
7. Bennett, D. A. (2001). How can I deal with missing data in my study? Australian and New Zealand Journal of Public Health, 25, 464–469.
8. Bernhard, J., Celia, D.F., &Coates, A.S. (1998). Missing quality of life data in cancer clinical trials: Serious problems and challenges. Statistics in Medicine, 17, 517-532.

9. Camili, G., & Shepard, L.A. (1994). Methods for identifying biased test items. London: Sage Publication.
10. Demir, E., & Parlak, B. (2012). Türkiye’de eğitim araştırmalarında kayıp veri sorunu. Journal of Measurement and Evaluation in Education and Psychology 3(1), 230-241.
11. Demir, E. (2013). Kayıp verilerin varlığında çoktan seçmeli testlerde madde ve test parametrelerinin kestirilmesi: SBS örneği [Item and test parameters estimations for multiple choice tests in the presence of missing data: The case of SBS]. Journal of Educational Sciences Research, 3(2), 47–68.
12. Dişçi, R. (2012). Temel ve klinik biyoistatistik. İstanbul: Tıp Kitapevi.
13. Doğan, N., & Öğretmen, T. (2008). Değişen Madde Fonksiyonunu belirlemede Mantel–Haenszel, Ki-Kare ve Lojistik Regresyon tekniklerinin karşılaştırılması. Education and Science, 33(148).
14. Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. London: Lawrence Erlbaum Associates.
15. Emenogu, B. C., Falenchuck, O., & Childs, R. A. (2010). The effect of missing data treatment on Mantel-Haenszel DIF detection. The Alberta Journal of Educational Research, 56(4), 459-469.
16. Falenchuk, O., & Herbert, M. (2009). Investigation of differential non-response as a factor affecting the results of Mantel-Haenszel DIF detection California: American Educational Research Association.
17. Finch, W.H. (2011). The impact of missing data on the detection of nonuniform differential ıtem functioning. Educational and Psychological Measurement, 71(4) 663–683.
18. Garrett, P. L. (2009). A monte carlo study investigating missing data, differential item functioning, and effect size. Georgia State University, Unpublished doctoral dissertation.
19. Gelin, M.N. & Zumbo, B.D. (2003). Differential item functioning results may change depending on how an item is scored: an illustration with the center for epidemiologic studies depression scale. Educational and Psychological Measurement, X(X) DOI: 10.1177/0013164402239317.
20. Gierl, M.J., Jodoin, M.G., & Ackerman, T.A. (2000). Performance of Mantel-Haenszel, Simultaneous Item Bias Test, and Logistic Regression when the proportion of DIF items is large. American Educational Research Association.
21. Gonzales, A., Padilla, J.L., Dolores, H., Gomez-Benito, J., & Benitez, I. (2010). EASY-DIF: Software for analyzing differential item functioning using the Mantel-Haenszel and Standardization procedures. Applied Psychological Measurement. doi:10.1177/0146621610381489.
22. Graham, J.W. (2009). Missing Data Analysis: Making it work in the real world. Annual Review of Psychology, 60(4), 549-576.
23. Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646-675.
24. Gözen Çıtak, G. (2007). Klasik test ve madde-tepki kuramlarına göre çoktan seçmeli testlerde farklı puanlama yöntemlerinin karşılaştırılması. Doktora Tezi, Ankara Üniversitesi, Ankara
25. Hambletton, R.K. & Swaminathan, H. (1985). Item Response Theory: Principles and applications. Boston: Kluwer-Nijhoff Publishing.
26. Harwell, M. Stone, C. A., Hsu, T.C., & Kirisci, L. (1996). Monte carlo studies in item response theory. Applied Psychological Measurement, 20, 101-125.
27. Hohensinn, C. & Kubinger K. D. (2011). On the impact of missing values on item fit and the model validness of the Rasch model. Psychological Test and Assessment Modeling, 53, 380-393.
28. Kan, A., Sünbül, Ö., Ömür, S. (2013). 6.- 8. sınıf seviye belirleme sınavları alt testlerinin çeşitli yöntemlere göre değişen madde fonksiyonlarının incelenmesi. Mersin University Journal of the Faculty of Education, 9(2), 207-222.
29. Kothari, C.R. (2004). Research methodology: Methods and techniques (Second Revised Edition). New Delhi: New Age Int. Ltd.
30. Kristanjansonn E., R. Aylesworth, I. McDowell & B.D. Zumbo (2005). A Comparison of four methods for detecting differential item functioning in ordered response model. Educational and Psychological Measurement. 65(6), 935-953.
31. Little, R. J. A & Rubin, D. B. (1987). Statistical analysis with missing data (2nd ed.). New York: John Wiley & Sons, Inc.
32. Lord, F. M. (1974). Estimation of latent ability and item parameters when there are omitted responses. Psychometrika, 39, 247-264.
33. Lord, F. M. (1980). Applications of item response theory to practical testing problems. New Jersey: Lawrence Erlbaum Associates.
34. Molenberghs, G., & Kenward, M.G. (2007). Missing data in clinical studie (1 st ed.). England: John Wiley&Sons.
35. Narayanan, P., & Swaminathan, H. (1994). Performance of the Mantel-Haenszel and Simultaneous Item Bias procedures for detecting differential ıtem functioning, Applied Psychological Measurement, 18(4).
36. Osterlind, S.J. (1983). Test item bias. London: Sage Publication.
37. Padilla, J.L., Hidalgo, J.L., Benitez, I., & Gomez-Benito, J. (2012). Comparison of three software programs for evaluating DIF by means of the Mantel-Haenszel procedure; EASY DIF, DIFAS and EZDIF, Psicologica, 33,135-156.
38. Peng, C.Y.J., Harwell, M., Liou, S.M., & Ehman, L. H. (2006). Advances in missing data methods and implications for educational research. In S. Sawilowsky (Ed.), Greenwich: Real data analysis.
39. Peng, C. J., & Zhu, J. (2008). Comparison of two approaches for handling missing covariates in logistic regression. Educational and Psychological Measurement, 68(1), 58-77.
40. Pigott, T.D. (2001). A review of methods for missing data. Educational Research and Evaluation, 7(4); 353-383.
41. Robitzsch, A, & Rupp, A.A. (2009). Impact of missing data on the detection of differential item functioning the case of mantel-haenszel and logistic regression analysis. Educational and Psychological Measurement, 69(1): 18-34.
42. Rousseau, M., Bertrand, R., & Boiteau, N. (2006, April). Impact of missing data treatment on the efficiency of DIF methods. California: National Council on Measurement in Education.
43. Royce, S., Straits, B.C., & Straits, M.M. (1993). Approaches to social research (2nd ed.). New York: Oxford University Press.
44. Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581-592.
45. Schafer, J. L. (1999). Multiple imputation: A primer. Statistical Methods in Medical Research, (8), 3-15.
46. Sedivy, S. K., Zhang, B., & Traxel, N. M. (2006). Detection of differential item functioning with polytomous items in the presence of missing data. California: National Council of Measurement in Education.
47. Selvi, H. (2013). Klasik test ve madde tepki kuramlarına dayalı değişen madde fonksiyonu belirleme tekniklerinin farklı puanlama durumlarında incelenmesi. Yayınlanmamış Doktora Tezi. Mersin Üniversitesi Eğitim Bilimleri Enstitüsü.
48. Singh, Y.K. (2006). Fundamental of research methodology and statistics. New Delhi: New Age Int. Ltd.
49. Spray, J., & Miller, T. (1994). Identifying nonuniform DIF in polytomously scored test items. American College Testing Research Report Series 94-1. Iowa City, IA: American College Testing Program.
50. Ward, W.C., & Bennett, R.E. (2012). Construction versus choice in cognitive measurement: issues in constructed response, performance testing, and portfolio assessment. London and New York: Routledge, Taylor & Francis Group.
51. Woodward, M., Smith, W.C., & Tunstall Pedoe H. (1991). Bias from missing values: Sex differences in implication of failed venepuncture for the Scottish Health Study.Int J. Epidemiol.
52. Wu, A. D., Li, Z., & Zumbo, B. D. (2007). Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analysis: A demonstration with TIMSS data. Practical Assessment, Research & Evaluation, 12(3), 1-26.
53. Zumbo, B. D. (1999). A Handbook on the theory and methods of Differential Item Functioning (DIF): Logistic Regression modeling as a unitary framework for binary and likert-type (ordinal) item scores. Ottawa ON: Directorate of Human Resources Research and Evaluation, Department of National Defense.