Use of Full Hierarchy Consistency Index to Assess Response Consistency

Year 2018, , 105 - 118, 01.01.2018

Lokman Akbay , Mustafa Kılınç

https://doi.org/10.21449/ijate.350499

Abstract

Measurement models need to properly delineate
the real aspect of examinees’ response processes for measurement accuracy
purposes. To avoid invalid inferences, fit of examinees’ response data to the
model is studied through person-fit
statistics. Misfit between the examinee response data and measurement model may
be due to invalid models and/or examinee’s aberrant response behavior such as
cheating, creative responding, and random responding. Hierarchy consistency
index (HCI) was introduced as a person-fit statistics to assess classification
reliability of particular cognitive diagnosis models. This study examines the
HCI in terms of its usefulness under nonhierarchical attribute conditions and
under different item types. Moreover, current form of HCI formulation only
considers the information based on correct answers only. We argue and
demonstrate that more information could be obtained by incorporating the
information that may be obtained from incorrect responses. Therefore, this
study considers the full-version of the HCI (i.e., FHCI). Results indicate that
current form of HCI is sensitive to misfitting item types (i.e., basic or more
complex) and examinee attribute patterns. In other words, HCI is affected by
the attribute pattern an examinee has as well as by the item s/he aberrantly
responded. Yet, FHCI is not severely affected by item types under any examinee
attribute pattern.

Keywords

Person-fit, Attribute Hierarchy Index, Cognitive Diagnosis

References

• Cui, Y., & Leighton, J. P. (2009). The hierarchy consistency index: Evaluating person fit for cognitive diagnostic assessment. Journal of Educational Measurement, 46, 429-449.
• de la Torre, J. (2009b). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115-130.
• Doornik, J. A. (2011). Object-oriented matrix programming using Ox (Version 6.20). London: Timberlake Consultants Press.
• Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.
• Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuokas rule-space approach. Journal of Educational Measurement, 41, 205-237.
• Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187-212.
• Robitzsch, A., Kiefer, T., George, A. C., & Uenlue, A. (2014). CDM: Cognitive diagnosis modeling. R package version 5.8-9. https://CRAN.R-project.org/package=CDM
• Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.
• Templin, J. L., & Bradshaw, L. (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79, 317-339.