Wald Test Formulations in DIF Detection of CDM Data with the Proportional Reasoning Test

Abstract

This study aims to conduct differential item functioning analyses in the context of cognitive diagnosis assessments using various formulations of the Wald test. In implementing the Wald test, two scenarios are considered: one where the underlying reduced model can be assumed; and another where a saturated CDM is used. Illustration of the different Wald test to detect DIF in CDM data was based on the items’ performance of the Proportional Reasoning test among low- and high-performing school students. A benchmark simulation study was included to compare the performance of the Wald test in each scenario. The agreement of the latent attribute classification based on different cognitive diagnosis models was also discussed.

Keywords

• Cognitive Diagnosis Model
• Proportional Reasoning
• Wald Test
• DIF

References

1. Camilli, G. (2006). Test fairness. Educational Measurement, 4, 221-256.
2. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.
3. de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76, 179-199.
4. de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353.
5. de la Torre, J., & Lee, Y. S. (2013). Evaluating the wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50, 355-373.
6. Doornik, J. A. (2002). An object-oriented matrix programming using Ox (Version 3.1) [Computer software]. London, UK: Timberlake Consultants Press.
7. Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301-321.
8. Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (p. 129–145). Hilldale, NJ: Lawrence Earlbaum Associates.

9. Hou, L., de la Torre, J., & Nandakumar, R. (2014). Differential item functioning assessment in cognitive diagnostic modeling: Application of the wald test to investigate DIF in the DINA model. Journal of Educational Measurement, 51, 98-125.
10. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.
11. Li, F. (2008). A modified higher-order DINA model for detecting differential item functioning and differential attribute functioning (Doctoral dissertation). University of Georgia, Athens, GA.
12. Ma, W., Iaconangelo, C., & de la Torre, J. (2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40, 200-217.
13. Milewski, G. B., & Baron, P. A. (2002). Extending DIF methods to inform aggregate reports on cognitive skills. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
14. Morrison, D. F. (1967). Multivariate statistical methods. New York, NY: McGraw-Hill.
15. Shealy, R., & Stout, W. (1993). A model-based standardization approach that separates true bias/dif from group ability differences and detects test bias/dtf as well as item bias/dif. Psychometrika, 58, 159-194.
16. Tatsuoka, K. K. (1984). Analysis of errors in fraction addition and subtraction problems. Computer-based Education Research Laboratory, University of Illinois.
17. Terzi, R. (2017). New Q-matrix validation procedures (Doctoral dissertation). Rutgers, The State University of New Jersey, New Brunswick, NJ.
18. Tjoe, H., & de la Torre, J. (2014). The identification and validation process of proportional reasoning attributes: An application of a cognitive diagnosis modeling framework. Mathematics Education Research Journal, 26, 237-255.
19. Zhang, W. (2006). Detecting differential item functioning using the DINA model (Doctoral dissertation). University of North Carolina at Greensboro, Greensboro, NC.
20. Zumbo, B. D. (2007). Three generations of dif analyses: Considering where it has been where it is now, and where it is going. Language Assessment Quarterly, 4, 223-233.