The Effect of Missing Data Handling Methods on Differential Item Functioning with Testlet Data

Abstract

This study examined the effect of three missing data handling methods (listwise deletion, zero imputation and fractional hot-deck imputation) on differential item functioning (DIF) with testlet data with a variety of sample size and missing data percentage under missing completely at random , missing at random, and missing not at random missing mechanisms. The study was conducted on two different datasets consisting of six testlets which contain 20 reading comprehension items of a foreign language test. Data with left-skewed distribution was referred to as data1 and data with right-skewed distribution was referred to as data2. In current study false DIF was identified in data1 with all missing data methods under the missing at random mechanism with a 5% missing data rate in small sample size. Similarly, in analyses performed under the missing at random mechanism for data2, the proportion of items classified as false DIF was notably higher in the small sample size. Results also indicated that in all conditions, list wise deletion had the lowest correlations with DIF values obtained from the original datasets, datasets containing no missing data and serve as a reference for comparative analyses with datasets where missing data were artificially introduced. The zero imputation and fractional hot-deck imputation methods produced similar correlations when the missing data percentage was set at 5%. However, in the case of 15% missing data, zero imputation exhibited higher correlation values. Besides, in all conditions correlation values decreased with the increase of missing data percentage regardless of the missing data handling method.

Keywords

• testlet
• missing data
• differential item functioning

References

1. References AERA, APA, and NCME (2014). The standards for educational and psychological testing. Washington, DC: American Psychological Association.
2. Akcan, R., & Atalay Kabasakal, K. (2023). The impact of missing data on the performances of DIF detection methods. Journal of Measurement and Evaluation in Education and Psychology, 14(1), 95-105. https://doi.org/10.21031/epod.1183617
3. Allison, P. D. (2002). Missing data. Thousand Oaks, CA, US: Sage publications.
4. Banks, K. (2015). An introduction to missing data in the context of differential item functioning. Practical Assessment, Research & Evaluation, 20(12), 1-10.
5. Banks, K., & Walker, C. (2006). Performance of SIBTEST when focal group examinees have missing data. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
6. Beretvas, S. N., & Walker, C. M. (2012). Distinguishing differential testlet functioning from differential bundle functioning using the multilevel measurement model. Educational and Psychological Measurement, 72(2), 200-223. https://doi.org/10.1177/0013164411412768
7. Chang, H.-H., Mazzeo, J., & Roussos, L. (1996). Detecting DIF for polytomously scored items: An adaptation of the SIBTEST procedure. ETS Research Report Series, 1995(1), i:30.
8. Clauser, B. E., & Mazor, K. M. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement Issues and Practice, 17(1), 31-44. https://eric.ed.gov/?id=EJ564712

9. DeMars, C. E. (2006). Application of the bi-factor multidimensional item response theory model to testlet-based tests. Journal of Educational Measurement, 43(2), 145-168. https://doi.org/10.1111/j.1745-3984.2006.00010.x
10. DeMars, C. E. (2012). Confirming testlet effects. Applied Psychological Measurement, 36(2), 104- 121. https://doi.org/10.1177/0146621612437403