puje

Pamukkale Üniversitesi Eğitim Fakültesi Dergisi

1301-0085 1309-0275

Pamukkale University

10.9779/pauefd.546797

The Effect of the Item–Attribute Relation on the DINA Model Estimations in the Presence of Missing Data

Kayıp Veri Varlığında DINA Model Madde-Özellik İlişkisinin Parametre Kestirimine Etkisi

https://orcid.org/0000-0001-7088-4268

Kalkan

Ömür Kaya

PAMUKKALE ÜNİVERSİTESİ, EĞİTİM FAKÜLTESİ

https://orcid.org/0000-0002-4821-0045

Başokçu

Tahsin Oğuz

EGE ÜNİVERSİTESİ, EĞİTİM FAKÜLTESİ

05 22 2019

46 46 290 306 03 29 2019 04 17 2019

1996

Pamukkale University Journal of Education

The objective of this studyis to investigate the relation between the number of items and attributes andto analyze the manner in which the different rates of missing data affect themodel estimations based on the simulation data. A Q-matrix contains 24 items,and data are generated using four attributes. A dataset of n = 3000 isgenerated by associating the first, middle, and final eight items in theQ-matrix with one, two, and three attributes, respectively, and 5%, 10%, and15% of the data have been randomly deleted from the first, middle, and finaleight-item blocks in the Q-matrix, respectively. Subsequently,imputation was performed using the multiple imputation (MI) method with thesedatasets, 100 replication was performed for each condition. The values obtainedfrom these datasets were compared with the values obtained from the fulldataset. Thus, it can be observed that an increase in the amount of missingdata negatively affects the consistency of the DINA parameters and the latentclass estimations. Further, the latent class consistency becomes less affectedby the missing data as the number of attributes associated with the itemsincrease. With an increase in the number of attributes associated with theitems, the missing data in these items affect the consistency level of the gparameter (guessing) less and the s parameter (slip) more. Furthermore, it canbe observed from the results that the test developers using the cognitivediagnosis models should specifically consider theitem–attribute relation in items with missing data.

Bu araştırmanınamacı, farklı oranlarda kayıp veri varlığında madde-özellik sayısı ilişkisinin,DINA model kestirimlerini nasıl etkilediğini incelediğini simülasyon verileriüzerinden incelemektir. Verilerin üretimlesinde dört özellik ve 24 maddedenoluşan bir Q matris kullanılmıştır. Q matrixteki ilk, orta ve son 8 maddesırasıyla 1, 2 ve 3 özellikle ilişkilendirilerek 3000 kişilik bir veri setiüretilmiş ve bu verilerde yer alan her 8 maddelik bloktan sırası ile %5, %10 ve%15 veri rassal silinmiştir. Ardından, bu veri setlerine MI yöntemi ileimputasyon yapılmıştır. Bu işlemler, her bir koşul için 100 keztekrarlanmıştır. Bu veri setlerinden elde edilen kestirimler, kayıpsız verisetinden elde edilen değerler ile karşılaştırılmıştır. Araştırmanın bulgularıkayıp veri miktarındaki artışın, DINA model parametre ve örtük sınıfkestirimlerindeki tutarlılığı olumsuz yönde etkilediğini göstermiştir. Maddeninilişkili olduğu özellik sayısı arttıkça örtük sınıf uyumu kayıp veriden daha azetkilenmiştir. Maddenin ilişkili olduğu attribute sayısı arttıkça bu maddelerdegözlenen kayıp veri, testin g parametresi uyum düzeyini daha az, sparametresini daha çok etkilemiştir. Araştırmanın sonuçları özellikle CDMmodellerini kullanan test geliştiricilerinin kayıp veri gözlenen maddelerde,madde-özellik ilişkisini göz önünde bulundurmaları gerektiğini göstermektedir.

DINA model missing data latent class estimates item–attribute relation

DINA model kayıp veri örtük sınıf kestirimi madde-özellik ilişkisi

Başokçu, T. O. , Kalkan, Ö. K., & Öğretmen, T. (2016, September). DINA Modele Dayalı Madde Parametre Kestiriminde Kayıp Veri Ele Alma Yöntemleri Etkisinin İncelenmesi. [An Investigation of the Effect of Missing Data Handling Methods on Parameter Estimation Based on DINA Model]. Paper presented at the Fifth International Congress on Measurement and Evaluation in Education and Psychology, Antalya, Turkey. p134-135. Abstract retrieved from http://epod2016.akdeniz.edu.tr/_dinamik/333/53.pdf

Bennett, D. A. (2001). How can i deal with missing data in my study? Australian and New Zealand journal of public health, 25(5), 464-469.

Buuren, S. V., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in R. Journal of statistical software, 45(3), 1-67.

Chen, J. (2017). A Residual-Based Approach to Validate Q-Matrix Specifications. Applied Psychological Measurement, (135), 014662161668602. https://doi.org/10.1177/0146621616686021de La Torre, J. (2008). An Empirically Based Method of Q‐Matrix Validation for the DINA Model: Development and Applications. Journal of educational measurement, 45(4), 343-362.

de La Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of educational and behavioral statistics, 34(1), 115-130.

de La Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179-199.

de la Torre, J., & Douglas, J. A. (2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data.Psychometrika, 73(4), 595-624.

de La Torre, J., Hong, Y., & Deng, W. (2010). Factors affecting the item parameter estimation and classification accuracy of the DINA model. Journal of Educational Measurement, 47(2), 227-249.

de la Torre, J., & Lee, Y. S. (2010). A Note on the Invariance of the DINA Model Parameters. Journal of Educational Measurement, 47(1), 115–127. https://doi.org/10.1111/j.1745-3984.2009.00102.x

Dong, Y., & Peng, C. Y. J. (2013). Principled missing data methods for researchers. SpringerPlus, 2(1), 222.

Doornik, J. A. (2018). An Object-Oriented Matrix Programming Language Ox 8. Timberlake Consultants.

Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah. NJ: Erlbaum.

Enders, C. K. (2010). Applied missing data analysis. Guilford Press.

Finch, H. (2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45(3), 225-245.

Fischer, G. H. (1973). The linear logistic test model as an instrument in educational research. Acta Psychologica, 37(6), 359–374. https://doi.org/10.1016/0001-6918(73)90003-6.

Graham, J. W., Taylor, B. J., Olchowski, A. E., & Cumsille, P. E. (2006). Planned missing data designs in psychological research. Psychological methods, 11(4), 323.

Haertel, E. H. (1989). Using Restricted Latent Class Models to Map the Skill of Achievement Structure Items. Journal of Educational Measurement, 26, 333–352. http://dx.doi.org/10.1111/j.1745-3984.1989.tb00336.x

Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, Department of Statistics, University of Illinois, Urbana-Champaign.

Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29(4), 262-277.

Henson, R. A., Templin, J. L., & Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191.

Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258-272.

Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data. Wiley. New York.

Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64(2), 187-212.

Nichols, P. D., Chipman, S. F., & Brennan, R. L. (2012). Cognitively diagnostic assessment. Routledge.Peugh, J. L., & Enders, C. K. (2004). Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of educational research, 74(4), 525-556.

R Core Team. (2018). R: A language and environment for statistical computing [Computer Software]. Vienna, Austria: R Foundation for Statistical Computing.

Robitzsch, A., Kiefer, T., George, A. C., Uenlue, A., & Robitzsch, M. A. (2018). Package ‘CDM’.

Rupp, A. A., & Mislevy, R. J. (2007). Cognitive Foundations of Structured Item Response Models. In Leighton, J., Gierl, M. (Eds.). Cognitive diagnostic assessment for education: Theory and applications, 205-241. New York: Cambridge University Press.

Rupp, A. A., & Templin, J. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78-96.

Schlomer, G. L., Bauman, S., & Card, N. A. (2010). Best practices for missing data management in counseling psychology. Journal of Counseling psychology, 57(1), 1.

Sen, S., & Bradshaw, L. (2017). Comparison of Relative Fit Indices for Diagnostic Model Selection. Applied Psychological Measurement, 014662161769552. https://doi.org/10.1177/0146621617695521

Sijtsma, K., & Van der Ark, L. A. (2003). Investigation and treatment of missing item scores in test and questionnaire data. Multivariate Behavioral Research, 38(4), 505-528.

Sorrel, M. A., Olea, J., Abad, F. J., de la Torre, J., Aguado, D., & Lievens, F. (2016). Validity and Reliability of Situational Judgement Test Scores. Organizational Research Methods, 19(3), 506–532. https://doi.org/10.1177/1094428116630065

Stekhoven, D. J. (2016). MissForest: nonparametric missing value imputation using random forest. R package version 1.4.

Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics. Allyn & Bacon/Pearson Education.

Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of educational measurement, 20(4), 345-354. http://dx.doi.org/10.1111/j.1745-3984.1983.tb00212.x

Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological methods, 11(3), 287.

van der Linden, W. J., & Hambleton, R. K. (2013). Item Response Theory: Brief History, Common Models, and Extensions. In Handbook of Modern Item Response Theory. https://doi.org/10.1007/978-1-4757-2691-6_1

von Davier, M. (2005). A general diagnostic model applied to language testing data (ETS Research Report RR-05-16). Princeton, NJ: Educational Testing Service.

Winship, C., Mare, R. D., & Warren, J. R. (2002). Latent class models for contingency tables with missing data. Applied latent class analysis, 408.

Zhang, B., & Walker, C. M. (2008). Impact of missing data on person—model fit and person trait estimation. Applied Psychological Measurement, 32(6), 466-479.