Effect of Q-Matrix Misspecification on Parameter Estimation in Differing Sample Sizes and Test Length for DINA

Abstract

-

Keywords

• -

Effect of Q-Matrix Misspecification on Parameter Estimation in Differing Sample Sizes and Test Length for DINA

Abstract

In Cognitive Diagnosis Models, every item in the measurement tool has a different effect (which is determined based on the attribute tested) on the classification of individuals in terms of attributes tested. One of the most effective factors that affects the quality of implications and the accuracy of classification, is to develop proper item‐attribute relationships, in other words, the correctness of Q‐matrix Misspecification of the Q‐matrix leads to incorrect decisions about the individuals. The present study, serving as a fundamental research, investigates the effect of the Q‐matrix misspecification in the DINA model on parameter estimations in the datasets, which are designed as a simulation and have differing sample sizes (50, 100, 250, 500, and 1,000 participants) and test length (15 and 30 items). The parameter estimations were made by using Markov Chain Monte Carlo method based on Bayesian estimation. The estimations for misspecified Q‐matrix have been compared to item parameters regarding the correct Q‐matrix appropriate to dataset. In the case of underspecification in Q‐matrix, slipping parameters for deficiently specified items and standard error values related to these; in the case of overspecification, guessing parameters related to overestimated items and standard error values related to these were overestimated. The parameter estimation is affected by the Q‐ matrix misspecification in all of the conditions discussed. Nevertheless, the amount of error in estimation does not show a regular differentiation in accordance with the sample size.

Keywords

• Misspecification, Guessing Parameter, Slipping Parameter

References

1. Baker, F. (1993) Sensitivity of the linear logistic test model to misspecification of the weight matrix. Applied Psychological Measurement, 17, 201‐210.
2. Best, N. G., Cowles, M. K., & Vines, K. (1995). CODA* convergence diagnosis and output analysis software for Gibbs sampling output Version 0.30. MRC Biostatistics Unit, Cambridge, 52.
3. Cassuto, N. (1996). The performance of the linear logistic test model under different testing conditions. (Doctoral dissertation). University of Minnesota, Minneapolis, MN.
4. Congdon, P. (2001). Applied Bayesian modelling. London, UK: John Wiley & Sons.
5. Culpepper, S. A. (2015). Bayesian estimation of the DINA model with Gibbs sampling. Journal of Educational and Behavioral Statistics. doi: 1076998615595403
6. De la Torre, J. (2008). An empirically‐based method of Q‐matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343‐362.
7. De la Torre, J. (2009a). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1), 115‐130.
8. De la Torre, J. (2009b). A cognitive diagnosis model for cognitively‐based multiple‐choice options. Applied Psychological Measurement, 33, 163‐183.

9. De la Torre, J., & Douglas, J. (2004). Higher‐order latent trait models for cognitive diagnosis. Psychometrika, 3(69), 333‐353.
10. De la Torre, J., Hong, Y., & Deng, W. (2010). Factors affecting the ıtem parameter estimation and classification accuracy of the DINA model. Journal of Educational Measurement, 47(2), 227‐249.
11. DiBello, L. V., & Stout, W. (2007). Guest editorsʹ ıntroduction and overview: IRT‐based cognitive diagnostic models and related methods. Journal of Educational Measurement, 44(4), 285‐291.
12. Dogan, E., & Tatsuoka, K. (2008). An international comparison using a diagnostic testing model: Turkish students’ profile of mathematical skills on TIMSS‐R. Educational Studies in Mathematics, 68(3), 263‐272.
13. Gelfand, A. E., Smith, A. F. M., & Lee, T‐M. (1992). Bayesian analysis of constrained parameter and truncated data problems using Gibbs sampling. Journal of the American Statistical Association, 87(418), 523‐532.
14. Gelfand, A. E. (2000). Gibbs sampling. Journal of the American Statistical Association, 95(452), 1300‐1304.
15. Gelman, A., & Rubin, D. R. (1992). A single series from the Gibbs sampler provides a false sense of security. In Bayesian statistics 4, 625‐631 (Ess: J. M. Bernardo et al.). Oxford: Oxford University Press.
16. Gill, J. (2002). Bayesian methods (A social and behavioral sciences approach). USA: Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences. Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333‐352.
17. Henson, R. (2004). Test discrimination and test construction for cognitive diagnostic models. (Doctoral dissertation). University of Illinois.
18. Henson, R., & Douglas, J. (2005) Test construction for cognitive diagnosis. Applied Psychological Measurement, 29(4), 262‐277.
19. Henson, R. A., & Templin, J. L. (2006). Implications of Q‐matrix misspecification in cognitive diagnosis. Manuscript submitted for publication