The Impact of Q-matrix Misspecification and Model Misuse on Classification Accuracy in the Generalized DINA Model

Abstract

This simulation study explored the impact of Q-matrix misspecification and model misuse on examinees’ classification accuracy within the generalized deterministic input, noisy “and” gate (G-DINA) model framework under the different conditions. The data was generated by saturated G-DINA model. Along with the generating model, two reduced models were used to fit the data: the additive CDM (A-CDM) and DINA model. The manipulated conditions included number of respondents, attribute correlations and test length. Two types of classification accuracy were examined: the overall classification accuracy and the class-specific classification accuracy. Results showed that the Q-matrix misspecification influenced classification accuracy more ominously than model misuse. The proportion of examinees classified correctly for each latent class was related to the types of Q-matrix misspecification. More test items had greater positive impact on classification accuracy than more respondents taking the test.

Keywords

• classification,cognitive diagnostic assessment,Q-matrix

Abstract

This simulation study investigated the impact of Q-matrix misspecification and model misuse on examinees’ classification accuracy within the generalized deterministic input, noisy “and” gate (G-DINA) model framework under various conditions. The data was generated in the saturated G-DINA model. Along with the generating model, two reduced models were used to fit the data: the additive CDM (A-CDM) and DINA model. The manipulated conditions included: number of respondents, attribute correlations and test length. Two types of classification accuracy were examined: the overall classification accuracy and the class-specific classification accuracy. Results showed that the Q-matrix misspecification influenced classification accuracy more ominously than model misuse. The proportion of examinees classified correctly for each latent class was related to the types of Q-matrix misspecification. More test items had greater positive impact on classification accuracy than more respondents taking the test.

Keywords

• classification,cognitive diagnostic assessment,Q-matrix

References

1. Henson, R., Roussos, L., Douglas, J., & He, X. (2008). Cognitive diagnostic attribute-level discrimination indices. Applied Psychological Measurement, 32(4), 275-288.