An investigation into the effect of different missing data imputation methods on IRT-based differential item functioning

Year 2024, , 445 - 462, 09.09.2024

Fatma Ünal , Hakan Koğar

https://doi.org/10.21449/ijate.1417166

Abstract

The purpose of this study is to examine the effect of missing data imputation methods, namely regression imputation (RI), multiple imputation (MI) and k-nearest neighbor (kNN) on differential item functioning (DIF). In this regard, the datasets used in the research were created by deleting some of the data via the missing completely at random mechanism from the complete datasets obtained from 600 students in Türkiye, the United Kingdom, the USA, New Zealand and Australia, who answered booklets 14 and 15 from the PISA 2018 science literacy test. Data imputation was applied to the datasets through missing data using RI, MI and kNN methods and DIF analysis was performed on all datasets in terms of language and gender variables via Lord’s χ2 method, Raju’s area measurement method and item response theory likelihood ratio method. DIF results from the complete datasets were taken as a reference and they were compared with the results from other datasets. As a result of the research, values close to 10% of accurate imputation were achieved in the RI method depending on language and gen-der variables. In MI and kNN methods, results closest to the complete datasets were obtained at a rate of 5% depending on the language variable. In the MI method, inaccurate results were obtained in all proportions in terms of the gender variable. For the gender variable, the kNN method gave accurate results at rates of 5% and 10%.

Keywords

Differential item functioning, Missing data, Item response theory, Raju’s area measurement, Likelihood ratio

References

• Altay, O. (2016). Genetik ve genetik olmayan faktörlere bağlı olarak Türk hastalarda varfarin dozajını tahmin eden bir uzman sistem geliştirilmesi [Improvement of an expert system that predict warfarin dosage in Turkish patients depending on genetic and non-genetic factors] [Master’s dissertation, Fırat University]. Higher Education Institution National Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/tezDetay.jsp?id=W663t01X1WehurHffLL0Q&no=Urx32Vn-YC2f6ufE0L3ZTw
• Atalay, K., Gök, B., Kelecioğlu, H., & Arsan, N. (2012). Değişen madde fonksiyonunun belirlenmesinde kullanılan farklı yöntemlerin karşılaştırılması: Bir simülasyon çalışması [Comparing different differential item functioning Methods: A simulation study]. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi (H. U. Journal of Education), 43, 270-281. https://dergipark.org.tr/tr/pub/hunefd/issue/7795/102030
• Atar, B., Atalay Kabasakal, K., Ünsal Özberk, E.B., Özberk, E.H., & Kıbrıslıoğlu Uysal, N., (2021). R ile veri analizi ve psikometri uygulamaları [Data analysis and psychometric applications with R] (3th ed.). Pegem Akademi.
• Baraldi, A.N., & Enders, C.K. (2010). An introduction to modern missing data analyses. Journal of school psychology, 48(1), 5 37. https://doi.org/10.1016/j.jsp.2009.10.001
• Başusta, N.B. (2013). PISA 2006 fen başarı testinin madde yanlılığının kültür ve dil açısından incelenmesi [Examination of item bias and language perspective of PISA 2006 science and culture achievement test] [Doctoral dissertation, Hacettepe University]. Hacettepe University Open Archive. https://www.openaccess.hacettepe.edu.tr/xmlui/bitstream/handle/11655/1766/42cc60c5 40f1 4b78 8c75 cc6d7932416e.pdf?sequence=1&isAllowed=y
• Bortolotti, S.L.V., Tezza, R., de Andrade, D.F., Bornia, A.C., & de Sousa Júnior, A.F. (2013). Relevance and advantages of using the item response theory. Quality & Quantity, 47, 2341 2360.
• Cihan, P. (2018). Veri madenciliği yöntemleriyle hayvan hastalıklarında teşhis, prognoz ve risk faktörlerinin belirlenmesi [Determination of dlagnosis, prognosis and risk factors inanimal diseases using by diseases using by data mining methods] [Doctoral dissertation, Yıldız Technical University]. Yıldız Technical University Open Archive. http://dspace.yildiz.edu.tr/xmlui/bitstream/handle/1/13155/7932.pdf?sequence=1&isAllowed=y
• Cromwell, S. (2002). A primer on ways to explore item bias. https://eric.ed.gov/?id=ED463307
• Çalışkan, S.K., & Soğukpınar, İ. (2008). Kxknn: K-means ve k en yakin komşu yöntemleri ile ağlarda nüfuz tespiti [Kxknn: Penetration detection in networks with k-means and k nearest neighbor methods]. EMO Yayınları, 120 24. https://www.emo.org.tr/ekler/8c1874c96244659_ek.pdf
• Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk, Ş. (2021). Sosyal bilimler için çok değişkenli istatistik: SPSS ve LISREL uygulamaları [Multivariate statistics for social sciences: SPSS and LISREL applications] (6th ed.). Pegem Akademi.
• Çüm, S., & Gelbal, S. (2015). Kayıp veriler yerine yaklaşık değer atamada kullanılan farklı yöntemlerin model veri uyumu üzerindeki etkisi [The effects of different methods used for value imputation instead of missing values on model data fit statistics]. Mehmet Akif Ersoy University Journal of Education Faculty, 1(35), 87 111. https://dergipark.org.tr/tr/pub/maeuefd/issue/19408/206357
• Çüm, S., Demir, E.K., Gelbal, S., & Kışla, T. (2018). Kayıp veriler yerine yaklaşık değer atamak için kullanılan gelişmiş yöntemlerin farklı koşullar altında karşılaştırılması [A comparison of advanced methods used for missing data imputation under different conditions]. Mehmet Akif Ersoy University Journal of Education Faculty, (45), 230-249. https://dergipark.org.tr/tr/pub/maeuefd/issue/35179/332605
• De Vellis, R.F. (2003). Scale development: Theory and applications. Applied Social Research Methods Series. Sage Publications, Inc. https://www.academia.edu/42875983/Scale_Developm_ent_Theory_and_Applications_Second_Edition
• Dogan, E., Guerrero, A., & Tatsuoka, K. (2005). Using DIF to investigate strengths and weaknesses in mathematics achievement profiles of 10 different countries. In annual meeting of the National Council on Measurement in Education (NCME), Montreal, Canada. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=a23cbcde509e6d6b9cd664236acc2d585b634578
• Dinçsoy, L.B. (2022). Karma testlerde kayıp verilerin değişen madde fonksiyonuna etkisinin incelenmesi [Investigation of the effect of missing data on differantial item functioning in mixed type tests] [Master’s dissertation, Hacettepe University]. Hacettepe University. https://openaccess.hacettepe.edu.tr/xmlui/bitstream/handle/11655/25949/10440993.pdf?sequence=1&isAllowed=y
• Emenogu, B.C., Falenchuk, O., & Childs, R.A. (2010). The effect of missing data treatment on Mantel-Haenszel DIF detection. The Alberta Journal of Educational Research, 56(4), 459- 469. https://doi.org/10.11575/ajer.v56i4.55429
• Enders, C.K. (2010). Applied missing data analysis (1th ed.). The Guilford Publications, Inc. http://hsta559s12.pbworks.com/w/file/fetch/52112520/enders.applied
• Erdoğan, K.H. (2019). Doğrulayıcı faktör analizi ve farklı veri setlerinde uygulanması [Confirmatory factory analysis and application to different datasets] [Master’s dissertation, Applied Sciences University of Isparta]. Higher Education Institution National Thesis Center. https://acikbilim.yok.gov.tr/bitstream/handle/20.500.12812/378756/yokAcikBilim_10284258.pdf?sequence=-1&isAllowed=y
• Garrett, P. (2009). A Monte Carlo study investigating missing data, differential item functioning, and effect size. Georgia State University. https://scholarworks.gsu.edu/cgi/viewcontent.cgi?article=1034&context=eps_diss
• Gök, B., Kabasakal, K.A., & Kelecioğlu, H. (2014). PISA 2009 öğrenci anketi tutum maddelerinin kültüre göre değişen madde fonksiyonu açısından incelenmesi [Analysis of attitude items in PISA2009 student questionnaire in terms of differential item functioning based on culture]. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 5(1), 72-87. https://doi.org/10.21031/epod.64124
• Gültekin, S., & Demirtaşlı, N.Ç. (2020). Comparing the test information obtained through multiple choice, open-ended and mixed item tests based on item response theory. Elementary Education Online, 11(1), 251 251. https://www.ilkogretim online.org/fulltext/218-1596943363.pdf?1697476130
• Hambleton, R.K., & Swaminathan, H. (2013). Item response theory: Principles and applications. Springer Science & Business Media.
• Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory (Vol. 2). Sage.
• Jabrayilov, R., Emons, W.H., & Sijtsma, K. (2016). Comparison of classical test theory and item response theory in individual change assessment. Applied Psychological Measurement, 40(8), 559 572. https://doi.org/10.1177/0146621616664046
• Jodoin, M.G., & Gierl, M.J. (2001). Evaluating type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329-349. https://eric.ed.gov/?id=EJ642273
• Josse, J., Mayer, I., Tierney, N., & Vialaneix, N. (2022). CRAN task view: Missing data. https://mirror.truenetwork.ru/CRAN/web/views/MissingData.html
• Kalaycıoğlu, D.B., & Kelecioğlu, H. (2011). Öğrenci Seçme Sınavı’nın madde yanlılığı açısından incelenmesi [Analysis of attitude items in PISA2009 student questionnaire in terms of differential item functioning based on culture]. Eğitim ve Bilim, 36(161), 3-13. http://egitimvebilim.ted.org.tr/index.php/EB/article/view/143/280
• Kim, S.H., Cohen, A.S., & Kim, H.O. (1994). An investigation of Lord’s procedure for the detection of differential item functioning. Applied Psychological Measurement, 18(3), 217-228. https://doi.org/10.1177/014662169401800303
• Longford, N.T. (2005). Missing data and small-area estimation: Modern analytical equipment for the survey statistician. Springer.
• Magis, D., Beland, S., Raiche, G., & Magis, M.D. (2015). Package ‘difR’. https://cran.r-project.org/web/packages/difR/difR.pdf
• MEB (2019). Uluslararası öğrenci değerlendirme programı PISA 2018 ulusal raporu [International student assessment program PISA 2018 national report]. Ankara: Directorate of Measurement, Evaluation and Testing Services, Ministry of National Education. https://www.meb.gov.tr/meb_iys_dosyalar/2019_12/03105347_pisa_2018_turkiye_on_raporu.pdf
• OECD (2019). PISA 2018 results volume I: What students know and can do. OECD Publishing. https://www.oecd.org/education/pisa-2018-results-volume-i-5f07c754-en.htm
• Peng, C.Y., Harwell, M.R., Liou, S.M., & Ehman, L.H. (2006). Advances in missing data methods and implications for educational research. In S. S. Sawilowsky (Ed.), Real Data Analysis (pp. 31-78).
• Raju, N.S. (1990). Determining the significance of estimated signed and unsigned areas between two item response functions. Applied Psychological Measurement, 14(2), 197-207. https://conservancy.umn.edu/bitstream/handle/11299/113559/v14n2p197.pdf?sequence=1 Rizopoulos, D., & Rizopoulos, M.D. (2018). Package ‘ltm’. https://cran.stat.unipd.it/web/packages/ltm/ltm.pdf
• Robitzsch, A., & Rupp, A.A. (2009). Impact of missing data on the detection of differential item functioning: The case of Mantel-Haenszel and logistic regression analysis. Educational and Psychological Measurement, 69(1), 18 34. https://doi.org/10.1177/0013164408318756
• Rogers, H.J., & Swaminathan, H. (1993). A comparison of logistic regression and Mantel Henszel procedures for detecting differential item functioning. Applied Psychological Measurement, 17(2), 105-116. https://doi.org/10.1177/014662169301700201
• Rosseel, Y., Jorgensen, T.D., Rockwood, N., Oberski, D., Byrnes, J., Vanbrabant, L., Savalei, V., Merkle, E., Hallquist, M., Rhemtulla, M., Katsikatsou, M., Barendse, M., Scharf, F., & Du, H. (June 17, 2017). Package ‘lavaan’. Version 0.6-18. https://cran.r-project.org/web/packages/lavaan/lavaan.pdf
• Salaria, N. (2012). Meaning of the term descriptive survey research method. International Journal of Transformations in Business Management, 1(6), 1 7.
• Schafer, J.L., & Graham, J.W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. https://doi.org/10.1037/1082-989X.7.2.147
• Schafer, J.L., & Olsen, M.K. (1998). Multiple imputation for multivariate missing- data problems: A data analyst’s perspective. Multivariate Behavioral Research, 33(4), 545-571. https://doi.org/10.1207/s15327906mbr3304_5
• Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23 74.
• Selvi, H., & Alıcı, D. (2018). Investigating the impact of missing data handling methods on the detection of differential item functioning. International Journal of Assessment Tools in Education, 5(1), 1-14. https://files.eric.ed.gov/fulltext/EJ1250131.pdf
• Sırgancı, G., & Çakan, M. (2020). Sıralı lojistik regresyon ve poly-sıbtest yöntemleri ile değişen madde fonksiyonunun belirlenmesi [Determination of the differential item function with ordered logistic regression and poly-sibtest methods]. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 20(1), 705 717. https://doi.org/10.17240/aibuefd.2020.20.52925-665084
• Sünbül, S.Ö., & Sünbül, Ö. (2016). Değişen madde fonksiyonunun belirlenmesinde kullanılan yöntemlerde I. Tip hata ve güç çalışması [Type I error rates and power study of several differential item functioning determination methods]. İlköğretim Online, 15(3), 882-897. https://doi.org/10.17051/io.2016.10640
• Tabachnick, B.G., & Fidell, L.S. (2013). Using multivariate statistics (6. Ed.). Pearson.
• Tamcı, P. (2018). Kayıp veriyle başa çıkma yöntemlerinin değişen madde fonksiyonu üzerindeki etkisinin incelenmesi [Investigation of the impact of techniques of handling missing data on differential item functioning] [Master’s dissertation, Hacettepe University]. Hacettepe University Open Archive. https://openaccess.hacettepe.edu.tr/xmlui/handle/11655/5315
• Taş, U.E., Arıcı, Ö., Ozarkan, H.B., & Özgürlük, B. (2016). PISA 2015 ulusal raporu [PISA 2015 national report]. Ministry of National Education. https://odsgm.meb.gov.tr/test/analizler/docs/PISA/PISA2015_Ulusal_Rapor.pdf
• Taşkıran, C., & Şenel, E. (2022). Çok boyutlu sportmenlik yönelimi ölçeğinin ölçme eşdeğerliğinin test edilmesi [Testing the measurement invariance of the multidimensional sportspersonship orientation scale]. International Journal of Sport Exercise and Training Sciences IJSETS, 8(4), 190 196. https://doi.org/10.18826/useeabd.1156699
• Templ, M., Alfons, A., Kowarik, A., Prantner, B., & Templ, M.M. (2016). VIM: Visualization and Imputation ofMissing Values. R package version 4.6.0, URL https://CRAN.R-project.org/package=VIM
• Thissen, D. (2001). IRTLRDIF v.2.0b: Software for the computation of the statistics involved in item response theory likelihood-ratio tests for differential item functioning. Chapel Hill: L.L. Thurstone
• Psychometric Laboratory, University of North Carolina at Chapel Hill.
• Uyar, Ş. (2015). Gözlenen gruplara ve örtük sınıflara göre tanımlananları madde etkilerinin karşılaştırılması [Comparing differential item functioning based on manifest groups and latent classes] [Doctoral dissertation, Hacettepe University]. Hacettepe University Open Access System. https://openaccess.hacettepe.edu.tr/xmlui/handle/11655/1816
• Van de Vijver, F.J., & Tanzer, N.K. (1997). Bias and equivalence in cross-cultural assessment: An overview. European Review of Applied Psychology, 47(4), 263-279. https://pure.uvt.nl/ws/files/225989/26727_11858.pdf
• Van Buuren, S. (2018). Flexible imputation of missing data. Chapman & Hall/CRC Press.
• Yılmaz, M. (2021). Eğilim puanları kullanılarak ABİDE çalışmasındaki maddelerin değişen madde fonksiyonu açısından incelenmesi [Investigation of differantial item functioning of the test items in the abide study by using propensity scores] [Master’s dissertation, Hacettepe University]. Hacettepe University Open Access System. https://openaccess.hacettepe.edu.tr/xmlui/handle/11655/23603