An investigation into the effect of different missing data imputation methods on IRT-based differential item functioning

Abstract

The purpose of this study is to examine the effect of missing data imputation methods, namely regression imputation (RI), multiple imputation (MI) and k-nearest neighbor (kNN) on differential item functioning (DIF). In this regard, the datasets used in the research were created by deleting some of the data via the missing completely at random mechanism from the complete datasets obtained from 600 students in Türkiye, the United Kingdom, the USA, New Zealand and Australia, who answered booklets 14 and 15 from the PISA 2018 science literacy test. Data imputation was applied to the datasets through missing data using RI, MI and kNN methods and DIF analysis was performed on all datasets in terms of language and gender variables via Lord’s χ2 method, Raju’s area measurement method and item response theory likelihood ratio method. DIF results from the complete datasets were taken as a reference and they were compared with the results from other datasets. As a result of the research, values close to 10% of accurate imputation were achieved in the RI method depending on language and gen-der variables. In MI and kNN methods, results closest to the complete datasets were obtained at a rate of 5% depending on the language variable. In the MI method, inaccurate results were obtained in all proportions in terms of the gender variable. For the gender variable, the kNN method gave accurate results at rates of 5% and 10%.

Keywords

• Differential item functioning
• Missing data
• Item response theory
• Raju’s area measurement
• Likelihood ratio

References

1. Altay, O. (2016). Genetik ve genetik olmayan faktörlere bağlı olarak Türk hastalarda varfarin dozajını tahmin eden bir uzman sistem geliştirilmesi [Improvement of an expert system that predict warfarin dosage in Turkish patients depending on genetic and non-genetic factors] [Master’s dissertation, Fırat University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezDetay.jsp?id=W663t01X1WehurHffLL0Q&no=Urx32Vn-YC2f6ufE0L3ZTw
2. Atalay, K., Gök, B., Kelecioğlu, H., & Arsan, N. (2012). Değişen madde fonksiyonunun belirlenmesinde kullanılan farklı yöntemlerin karşılaştırılması: Bir simülasyon çalışması [Comparing different differential item functioning Methods: A simulation study]. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi (H. U. Journal of Education), 43, 270-281. https://dergipark.org.tr/tr/pub/hunefd/issue/7795/102030
3. Atar, B., Atalay Kabasakal, K., Ünsal Özberk, E.B., Özberk, E.H., & Kıbrıslıoğlu Uysal, N., (2021). R ile veri analizi ve psikometri uygulamaları [Data analysis and psychometric applications with R] (3th ed.). Pegem Akademi.
4. Baraldi, A.N., & Enders, C.K. (2010). An introduction to modern missing data analyses. Journal of school psychology, 48(1), 5 37. https://doi.org/10.1016/j.jsp.2009.10.001
5. Başusta, N.B. (2013). PISA 2006 fen başarı testinin madde yanlılığının kültür ve dil açısından incelenmesi [Examination of item bias and language perspective of PISA 2006 science and culture achievement test] [Doctoral dissertation, Hacettepe University]. Hacettepe University Open Archive. https://www.openaccess.hacettepe.edu.tr/xmlui/bitstream/handle/11655/1766/42cc60c5 40f1 4b78 8c75 cc6d7932416e.pdf?sequence=1&isAllowed=y
6. Bortolotti, S.L.V., Tezza, R., de Andrade, D.F., Bornia, A.C., & de Sousa Júnior, A.F. (2013). Relevance and advantages of using the item response theory. Quality & Quantity, 47, 2341 2360.
7. Cihan, P. (2018). Veri madenciliği yöntemleriyle hayvan hastalıklarında teşhis, prognoz ve risk faktörlerinin belirlenmesi [Determination of dlagnosis, prognosis and risk factors inanimal diseases using by diseases using by data mining methods] [Doctoral dissertation, Yıldız Technical University]. Yıldız Technical University Open Archive. http://dspace.yildiz.edu.tr/xmlui/bitstream/handle/1/13155/7932.pdf?sequence=1&isAllowed=y
8. Cromwell, S. (2002). A primer on ways to explore item bias. https://eric.ed.gov/?id=ED463307

9. Çalışkan, S.K., & Soğukpınar, İ. (2008). Kxknn: K-means ve k en yakin komşu yöntemleri ile ağlarda nüfuz tespiti [Kxknn: Penetration detection in networks with k-means and k nearest neighbor methods]. EMO Yayınları, 120 24. https://www.emo.org.tr/ekler/8c1874c96244659_ek.pdf
10. Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk, Ş. (2021). Sosyal bilimler için çok değişkenli istatistik: SPSS ve LISREL uygulamaları [Multivariate statistics for social sciences: SPSS and LISREL applications] (6th ed.). Pegem Akademi.
11. Çüm, S., & Gelbal, S. (2015). Kayıp veriler yerine yaklaşık değer atamada kullanılan farklı yöntemlerin model veri uyumu üzerindeki etkisi [The effects of different methods used for value imputation instead of missing values on model data fit statistics]. Mehmet Akif Ersoy University Journal of Education Faculty, 1(35), 87 111. https://dergipark.org.tr/tr/pub/maeuefd/issue/19408/206357
12. Çüm, S., Demir, E.K., Gelbal, S., & Kışla, T. (2018). Kayıp veriler yerine yaklaşık değer atamak için kullanılan gelişmiş yöntemlerin farklı koşullar altında karşılaştırılması [A comparison of advanced methods used for missing data imputation under different conditions]. Mehmet Akif Ersoy University Journal of Education Faculty, (45), 230-249. https://dergipark.org.tr/tr/pub/maeuefd/issue/35179/332605
13. De Vellis, R.F. (2003). Scale development: Theory and applications. Applied Social Research Methods Series. Sage Publications, Inc. https://www.academia.edu/42875983/Scale_Developm_ent_Theory_and_Applications_Second_Edition
14. Dogan, E., Guerrero, A., & Tatsuoka, K. (2005). Using DIF to investigate strengths and weaknesses in mathematics achievement profiles of 10 different countries. In annual meeting of the National Council on Measurement in Education (NCME), Montreal, Canada. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=a23cbcde509e6d6b9cd664236acc2d585b634578
15. Dinçsoy, L.B. (2022). Karma testlerde kayıp verilerin değişen madde fonksiyonuna etkisinin incelenmesi [Investigation of the effect of missing data on differantial item functioning in mixed type tests] [Master’s dissertation, Hacettepe University]. Hacettepe University. https://openaccess.hacettepe.edu.tr/xmlui/bitstream/handle/11655/25949/10440993.pdf?sequence=1&isAllowed=y
16. Emenogu, B.C., Falenchuk, O., & Childs, R.A. (2010). The effect of missing data treatment on Mantel-Haenszel DIF detection. The Alberta Journal of Educational Research, 56(4), 459- 469. https://doi.org/10.11575/ajer.v56i4.55429
17. Enders, C.K. (2010). Applied missing data analysis (1th ed.). The Guilford Publications, Inc. http://hsta559s12.pbworks.com/w/file/fetch/52112520/enders.applied
18. Erdoğan, K.H. (2019). Doğrulayıcı faktör analizi ve farklı veri setlerinde uygulanması [Confirmatory factory analysis and application to different datasets] [Master’s dissertation, Applied Sciences University of Isparta]. Higher Education Institution National Thesis Center. https://acikbilim.yok.gov.tr/bitstream/handle/20.500.12812/378756/yokAcikBilim_10284258.pdf?sequence=-1&isAllowed=y
19. Garrett, P. (2009). A Monte Carlo study investigating missing data, differential item functioning, and effect size. Georgia State University. https://scholarworks.gsu.edu/cgi/viewcontent.cgi?article=1034&context=eps_diss
20. Gök, B., Kabasakal, K.A., & Kelecioğlu, H. (2014). PISA 2009 öğrenci anketi tutum maddelerinin kültüre göre değişen madde fonksiyonu açısından incelenmesi [Analysis of attitude items in PISA2009 student questionnaire in terms of differential item functioning based on culture]. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 5(1), 72-87. https://doi.org/10.21031/epod.64124
21. Gültekin, S., & Demirtaşlı, N.Ç. (2020). Comparing the test information obtained through multiple choice, open-ended and mixed item tests based on item response theory. Elementary Education Online, 11(1), 251 251. https://www.ilkogretim online.org/fulltext/218-1596943363.pdf?1697476130
22. Hambleton, R.K., & Swaminathan, H. (2013). Item response theory: Principles and applications. Springer Science & Business Media.
23. Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory (Vol. 2). Sage.
24. Jabrayilov, R., Emons, W.H., & Sijtsma, K. (2016). Comparison of classical test theory and item response theory in individual change assessment. Applied Psychological Measurement, 40(8), 559 572. https://doi.org/10.1177/0146621616664046
25. Jodoin, M.G., & Gierl, M.J. (2001). Evaluating type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329-349. https://eric.ed.gov/?id=EJ642273
26. Josse, J., Mayer, I., Tierney, N., & Vialaneix, N. (2022). CRAN task view: Missing data. https://mirror.truenetwork.ru/CRAN/web/views/MissingData.html
27. Kalaycıoğlu, D.B., & Kelecioğlu, H. (2011). Öğrenci Seçme Sınavı’nın madde yanlılığı açısından incelenmesi [Analysis of attitude items in PISA2009 student questionnaire in terms of differential item functioning based on culture]. Eğitim ve Bilim, 36(161), 3-13. http://egitimvebilim.ted.org.tr/index.php/EB/article/view/143/280
28. Kim, S.H., Cohen, A.S., & Kim, H.O. (1994). An investigation of Lord’s procedure for the detection of differential item functioning. Applied Psychological Measurement, 18(3), 217-228. https://doi.org/10.1177/014662169401800303
29. Longford, N.T. (2005). Missing data and small-area estimation: Modern analytical equipment for the survey statistician. Springer.
30. Magis, D., Beland, S., Raiche, G., & Magis, M.D. (2015). Package ‘difR’. https://cran.r-project.org/web/packages/difR/difR.pdf
31. MEB (2019). Uluslararası öğrenci değerlendirme programı PISA 2018 ulusal raporu [International student assessment program PISA 2018 national report]. Ankara: Directorate of Measurement, Evaluation and Testing Services, Ministry of National Education. https://www.meb.gov.tr/meb_iys_dosyalar/2019_12/03105347_pisa_2018_turkiye_on_raporu.pdf
32. OECD (2019). PISA 2018 results volume I: What students know and can do. OECD Publishing. https://www.oecd.org/education/pisa-2018-results-volume-i-5f07c754-en.htm
33. Peng, C.Y., Harwell, M.R., Liou, S.M., & Ehman, L.H. (2006). Advances in missing data methods and implications for educational research. In S. S. Sawilowsky (Ed.), Real Data Analysis (pp. 31-78).
34. Raju, N.S. (1990). Determining the significance of estimated signed and unsigned areas between two item response functions. Applied Psychological Measurement, 14(2), 197-207. https://conservancy.umn.edu/bitstream/handle/11299/113559/v14n2p197.pdf?sequence=1 Rizopoulos, D., & Rizopoulos, M.D. (2018). Package ‘ltm’. https://cran.stat.unipd.it/web/packages/ltm/ltm.pdf
35. Robitzsch, A., & Rupp, A.A. (2009). Impact of missing data on the detection of differential item functioning: The case of Mantel-Haenszel and logistic regression analysis. Educational and Psychological Measurement, 69(1), 18 34. https://doi.org/10.1177/0013164408318756
36. Rogers, H.J., & Swaminathan, H. (1993). A comparison of logistic regression and Mantel Henszel procedures for detecting differential item functioning. Applied Psychological Measurement, 17(2), 105-116. https://doi.org/10.1177/014662169301700201
37. Rosseel, Y., Jorgensen, T.D., Rockwood, N., Oberski, D., Byrnes, J., Vanbrabant, L., Savalei, V., Merkle, E., Hallquist, M., Rhemtulla, M., Katsikatsou, M., Barendse, M., Scharf, F., & Du, H. (June 17, 2017). Package ‘lavaan’. Version 0.6-18. https://cran.r-project.org/web/packages/lavaan/lavaan.pdf
38. Salaria, N. (2012). Meaning of the term descriptive survey research method. International Journal of Transformations in Business Management, 1(6), 1 7.
39. Schafer, J.L., & Graham, J.W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. https://doi.org/10.1037/1082-989X.7.2.147
40. Schafer, J.L., & Olsen, M.K. (1998). Multiple imputation for multivariate missing- data problems: A data analyst’s perspective. Multivariate Behavioral Research, 33(4), 545-571. https://doi.org/10.1207/s15327906mbr3304_5
41. Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23 74.
42. Selvi, H., & Alıcı, D. (2018). Investigating the impact of missing data handling methods on the detection of differential item functioning. International Journal of Assessment Tools in Education, 5(1), 1-14. https://files.eric.ed.gov/fulltext/EJ1250131.pdf
43. Sırgancı, G., & Çakan, M. (2020). Sıralı lojistik regresyon ve poly-sıbtest yöntemleri ile değişen madde fonksiyonunun belirlenmesi [Determination of the differential item function with ordered logistic regression and poly-sibtest methods]. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 20(1), 705 717. https://doi.org/10.17240/aibuefd.2020.20.52925-665084
44. Sünbül, S.Ö., & Sünbül, Ö. (2016). Değişen madde fonksiyonunun belirlenmesinde kullanılan yöntemlerde I. Tip hata ve güç çalışması [Type I error rates and power study of several differential item functioning determination methods]. İlköğretim Online, 15(3), 882-897. https://doi.org/10.17051/io.2016.10640
45. Tabachnick, B.G., & Fidell, L.S. (2013). Using multivariate statistics (6. Ed.). Pearson.
46. Tamcı, P. (2018). Kayıp veriyle başa çıkma yöntemlerinin değişen madde fonksiyonu üzerindeki etkisinin incelenmesi [Investigation of the impact of techniques of handling missing data on differential item functioning] [Master’s dissertation, Hacettepe University]. Hacettepe University Open Archive. https://openaccess.hacettepe.edu.tr/xmlui/handle/11655/5315
47. Taş, U.E., Arıcı, Ö., Ozarkan, H.B., & Özgürlük, B. (2016). PISA 2015 ulusal raporu [PISA 2015 national report]. Ministry of National Education. https://odsgm.meb.gov.tr/test/analizler/docs/PISA/PISA2015_Ulusal_Rapor.pdf
48. Taşkıran, C., & Şenel, E. (2022). Çok boyutlu sportmenlik yönelimi ölçeğinin ölçme eşdeğerliğinin test edilmesi [Testing the measurement invariance of the multidimensional sportspersonship orientation scale]. International Journal of Sport Exercise and Training Sciences IJSETS, 8(4), 190 196. https://doi.org/10.18826/useeabd.1156699
49. Templ, M., Alfons, A., Kowarik, A., Prantner, B., & Templ, M.M. (2016). VIM: Visualization and Imputation ofMissing Values. R package version 4.6.0, URL https://CRAN.R-project.org/package=VIM
50. Thissen, D. (2001). IRTLRDIF v.2.0b: Software for the computation of the statistics involved in item response theory likelihood-ratio tests for differential item functioning. Chapel Hill: L.L. Thurstone
51. Psychometric Laboratory, University of North Carolina at Chapel Hill.
52. Uyar, Ş. (2015). Gözlenen gruplara ve örtük sınıflara göre tanımlananları madde etkilerinin karşılaştırılması [Comparing differential item functioning based on manifest groups and latent classes] [Doctoral dissertation, Hacettepe University]. Hacettepe University Open Access System. https://openaccess.hacettepe.edu.tr/xmlui/handle/11655/1816
53. Van de Vijver, F.J., & Tanzer, N.K. (1997). Bias and equivalence in cross-cultural assessment: An overview. European Review of Applied Psychology, 47(4), 263-279. https://pure.uvt.nl/ws/files/225989/26727_11858.pdf
54. Van Buuren, S. (2018). Flexible imputation of missing data. Chapman & Hall/CRC Press.
55. Yılmaz, M. (2021). Eğilim puanları kullanılarak ABİDE çalışmasındaki maddelerin değişen madde fonksiyonu açısından incelenmesi [Investigation of differantial item functioning of the test items in the abide study by using propensity scores] [Master’s dissertation, Hacettepe University]. Hacettepe University Open Access System. https://openaccess.hacettepe.edu.tr/xmlui/handle/11655/23603