The Effect of the Item–Attribute Relation on the DINA Model Estimations in the Presence of Missing Data

Year 2019, Volume: 46 Issue: 46, 290 - 306, 22.05.2019

Ömür Kaya Kalkan , Tahsin Oğuz Başokçu

https://doi.org/10.9779/pauefd.546797

Abstract

The objective of this study
is to investigate the relation between the number of items and attributes and
to analyze the manner in which the different rates of missing data affect the
model estimations based on the simulation data. A Q-matrix contains 24 items,
and data are generated using four attributes. A dataset of n = 3000 is
generated by associating the first, middle, and final eight items in the
Q-matrix with one, two, and three attributes, respectively, and 5%, 10%, and
15% of the data have been randomly deleted from the first, middle, and final
eight-item blocks in the Q-matrix, respectively. Subsequently,
imputation was performed using the multiple imputation (MI) method with these
datasets, 100 replication was performed for each condition. The values obtained
from these datasets were compared with the values obtained from the full
dataset. Thus, it can be observed that an increase in the amount of missing
data negatively affects the consistency of the DINA parameters and the latent
class estimations. Further, the latent class consistency becomes less affected
by the missing data as the number of attributes associated with the items
increase. With an increase in the number of attributes associated with the
items, the missing data in these items affect the consistency level of the g
parameter (guessing) less and the s parameter (slip) more. Furthermore, it can
be observed from the results that the test developers using the cognitive
diagnosis models should specifically consider the
item–attribute relation in items with missing data.

Keywords

DINA model, missing data, latent class estimates, item–attribute relation

Kayıp Veri Varlığında DINA Model Madde-Özellik İlişkisinin Parametre Kestirimine Etkisi

Abstract

Bu araştırmanın
amacı, farklı oranlarda kayıp veri varlığında madde-özellik sayısı ilişkisinin,
DINA model kestirimlerini nasıl etkilediğini incelediğini simülasyon verileri
üzerinden incelemektir. Verilerin üretimlesinde dört özellik ve 24 maddeden
oluşan bir Q matris kullanılmıştır. Q matrixteki ilk, orta ve son 8 madde
sırasıyla 1, 2 ve 3 özellikle ilişkilendirilerek 3000 kişilik bir veri seti
üretilmiş ve bu verilerde yer alan her 8 maddelik bloktan sırası ile %5, %10 ve
%15 veri rassal silinmiştir. Ardından, bu veri setlerine MI yöntemi ile
imputasyon yapılmıştır. Bu işlemler, her bir koşul için 100 kez
tekrarlanmıştır. Bu veri setlerinden elde edilen kestirimler, kayıpsız veri
setinden elde edilen değerler ile karşılaştırılmıştır. Araştırmanın bulguları
kayıp veri miktarındaki artışın, DINA model parametre ve örtük sınıf
kestirimlerindeki tutarlılığı olumsuz yönde etkilediğini göstermiştir. Maddenin
ilişkili olduğu özellik sayısı arttıkça örtük sınıf uyumu kayıp veriden daha az
etkilenmiştir. Maddenin ilişkili olduğu attribute sayısı arttıkça bu maddelerde
gözlenen kayıp veri, testin g parametresi uyum düzeyini daha az, s
parametresini daha çok etkilemiştir. Araştırmanın sonuçları özellikle CDM
modellerini kullanan test geliştiricilerinin kayıp veri gözlenen maddelerde,
madde-özellik ilişkisini göz önünde bulundurmaları gerektiğini göstermektedir.

 

Keywords

DINA model, kayıp veri, örtük sınıf kestirimi, madde-özellik ilişkisi

References

• Başokçu, T. O. , Kalkan, Ö. K., & Öğretmen, T. (2016, September). DINA Modele Dayalı Madde Parametre Kestiriminde Kayıp Veri Ele Alma Yöntemleri Etkisinin İncelenmesi. [An Investigation of the Effect of Missing Data Handling Methods on Parameter Estimation Based on DINA Model]. Paper presented at the Fifth International Congress on Measurement and Evaluation in Education and Psychology, Antalya, Turkey. p134-135. Abstract retrieved from http://epod2016.akdeniz.edu.tr/_dinamik/333/53.pdf
• Bennett, D. A. (2001). How can i deal with missing data in my study? Australian and New Zealand journal of public health, 25(5), 464-469.
• Buuren, S. V., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in R. Journal of statistical software, 45(3), 1-67.
• Chen, J. (2017). A Residual-Based Approach to Validate Q-Matrix Specifications. Applied Psychological Measurement, (135), 014662161668602. https://doi.org/10.1177/0146621616686021de La Torre, J. (2008). An Empirically Based Method of Q‐Matrix Validation for the DINA Model: Development and Applications. Journal of educational measurement, 45(4), 343-362.
• de La Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of educational and behavioral statistics, 34(1), 115-130.
• de La Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179-199.
• de la Torre, J., & Douglas, J. A. (2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data.Psychometrika, 73(4), 595-624.
• de La Torre, J., Hong, Y., & Deng, W. (2010). Factors affecting the item parameter estimation and classification accuracy of the DINA model. Journal of Educational Measurement, 47(2), 227-249.
• de la Torre, J., & Lee, Y. S. (2010). A Note on the Invariance of the DINA Model Parameters. Journal of Educational Measurement, 47(1), 115–127. https://doi.org/10.1111/j.1745-3984.2009.00102.x
• Dong, Y., & Peng, C. Y. J. (2013). Principled missing data methods for researchers. SpringerPlus, 2(1), 222.
• Doornik, J. A. (2018). An Object-Oriented Matrix Programming Language Ox 8. Timberlake Consultants.
• Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah. NJ: Erlbaum.
• Enders, C. K. (2010). Applied missing data analysis. Guilford Press.
• Finch, H. (2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45(3), 225-245.
• Fischer, G. H. (1973). The linear logistic test model as an instrument in educational research. Acta Psychologica, 37(6), 359–374. https://doi.org/10.1016/0001-6918(73)90003-6.
• Graham, J. W., Taylor, B. J., Olchowski, A. E., & Cumsille, P. E. (2006). Planned missing data designs in psychological research. Psychological methods, 11(4), 323.
• Haertel, E. H. (1989). Using Restricted Latent Class Models to Map the Skill of Achievement Structure Items. Journal of Educational Measurement, 26, 333–352. http://dx.doi.org/10.1111/j.1745-3984.1989.tb00336.x
• Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, Department of Statistics, University of Illinois, Urbana-Champaign.
• Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29(4), 262-277.
• Henson, R. A., Templin, J. L., & Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191.
• Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258-272.
• Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data. Wiley. New York.
• Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64(2), 187-212.
• Nichols, P. D., Chipman, S. F., & Brennan, R. L. (2012). Cognitively diagnostic assessment. Routledge.Peugh, J. L., & Enders, C. K. (2004). Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of educational research, 74(4), 525-556.
• R Core Team. (2018). R: A language and environment for statistical computing [Computer Software]. Vienna, Austria: R Foundation for Statistical Computing.
• Robitzsch, A., Kiefer, T., George, A. C., Uenlue, A., & Robitzsch, M. A. (2018). Package ‘CDM’.
• Rupp, A. A., & Mislevy, R. J. (2007). Cognitive Foundations of Structured Item Response Models. In Leighton, J., Gierl, M. (Eds.). Cognitive diagnostic assessment for education: Theory and applications, 205-241. New York: Cambridge University Press.
• Rupp, A. A., & Templin, J. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78-96.
• Schlomer, G. L., Bauman, S., & Card, N. A. (2010). Best practices for missing data management in counseling psychology. Journal of Counseling psychology, 57(1), 1.
• Sen, S., & Bradshaw, L. (2017). Comparison of Relative Fit Indices for Diagnostic Model Selection. Applied Psychological Measurement, 014662161769552. https://doi.org/10.1177/0146621617695521
• Sijtsma, K., & Van der Ark, L. A. (2003). Investigation and treatment of missing item scores in test and questionnaire data. Multivariate Behavioral Research, 38(4), 505-528.
• Sorrel, M. A., Olea, J., Abad, F. J., de la Torre, J., Aguado, D., & Lievens, F. (2016). Validity and Reliability of Situational Judgement Test Scores. Organizational Research Methods, 19(3), 506–532. https://doi.org/10.1177/1094428116630065
• Stekhoven, D. J. (2016). MissForest: nonparametric missing value imputation using random forest. R package version 1.4.
• Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics. Allyn & Bacon/Pearson Education.
• Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of educational measurement, 20(4), 345-354. http://dx.doi.org/10.1111/j.1745-3984.1983.tb00212.x
• Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological methods, 11(3), 287.
• van der Linden, W. J., & Hambleton, R. K. (2013). Item Response Theory: Brief History, Common Models, and Extensions. In Handbook of Modern Item Response Theory. https://doi.org/10.1007/978-1-4757-2691-6_1
• von Davier, M. (2005). A general diagnostic model applied to language testing data (ETS Research Report RR-05-16). Princeton, NJ: Educational Testing Service.
• Winship, C., Mare, R. D., & Warren, J. R. (2002). Latent class models for contingency tables with missing data. Applied latent class analysis, 408.
• Zhang, B., & Walker, C. M. (2008). Impact of missing data on person—model fit and person trait estimation. Applied Psychological Measurement, 32(6), 466-479.