Where the Rivers Merge: Cognitive Diagnostic Approaches to Educational Assessment

Abstract

A growing emphasis on the union of cognitive psychology with psychometrics has led to the inception of Cognitive Diagnostic Assessment (CDA). CDA can be  defined as a cognitively‑grounded assessment methodology which aims to detect  examinees’ strengths and weaknesses in a given domain, make reliable diagnostic classifications directly from the statistical models, and present stakeholders with fine‑grained and pedagogically‑meaningful diagnostic feedback. Although CDA holds great promise for educational assessment practices, it remains relatively unknown to many assessment specialists. Hence, this paper aims to describe the theoretical underpinnings and working principles of CDA by giving information about the developments that have led to the inception of CDA and elaborate on how CDA can be implemented in operational assessment settings either by using an inductive or retrofitted approach to foster learning and enhance accountability within educational programs. Finally, the potential that CDA bears for educational assessment is discussed and practical implications are made. 

Keywords

• educational assessment,cognitive diagnostic assessment,diagnostic classification modelling,skills diagnosis

Eğitimde Ölçme ve Değerlendirme Uygulamalarına Bilişsel Tanılayıcı Bir Yaklaşım

Abstract

Bilişsel psikolojinin psikometri ile harmanlanması Bilişsel Tanılayıcı Değerlendirme (BTD) adı verilen ölçme ve değerlendirme yaklaşımının ortaya çıkmasını sağlamıştır. BTD, bilişsel temelli, istatistiki açıdan sofistike ve alternatif bir ölçme ve değerlendirme yaklaşımıdır. Bireylerin belirli bir beceri ya da akademik alandaki güçlü ve zayıf yanlarının, eksiklerinin ve yanılgılarının saptanmasını ve bu hususlara yönelik paydaşlara (öğrenci, öğretmen, veli ve idarecilere) bireylerin halihazırdaki durumları hakkında detaylı dönüt verilmesini amaçlar. Sağlanan dönüt, pedagojik açıdan anlamlı ve öğrenme sürecini destekleyici boyutta olmalıdır. Bu değerlendirme yaklaşımının eğitim öğretim faaliyetleri için pek çok yararı olmasına karşın, BTD hem eğitim araştırmacıları hem de ölçme değerlendirme alanında çalışan araştırmacılar tarafından yeteri derecede tanınmamaktadır. Bu makalede, BTD yaklaşımının ortaya çıkmasına sebep veren eğitimsel akım ve gelişmeler ele alınmış, BTD’nin kuramsal temelleri, çalışma prensipleri, işlevleri hakkında detaylı bilgi verilmiştir. Ayrıca, BTD’nin öğrenme çıktılarını iyileştirme ve eğitim programlarının kalite ve hesap verebilirliğinin artırılması hedeflerine yönelik olarak, eğitim ve ölçme değerlendirme ortamlarında nasıl uygulanabileceği hususunda öneriler sunulmuştur.

Keywords

• eğitimde ölçme ve değerlendirme,bilişsel tanılayıcı değerlendirme,tanılayıcı sınıflama modellemesi

References

1. Anastasi, A. (1967). Psychology, psychologists, and psychological testing. American Psychologist, 22, 297-306.
2. Beaton, A. E., & Allen, N. L. (1992). Interpreting scales through scale anchoring. Journal of Educational Statistics, 17, 191-204.
3. Bradshaw, L., Izsak, A., Templin, J., & Jacobson, E. (2014). Diagnosing teachers’ understandings of rational numbers: building a multidimensional test within the diagnostic classification framework. Educational Measurement: Issues and Practice, 33, 2-14.
4. Buck, G., Tatsuoka, K., & Kostin, I. (1997). The subskills of reading: Rule-space analysis of a multiple-choice test of second language reading comprehension. Language Learning, 47(3), 423-466.
5. Buck, G., & Tatsuoka, K. (1998). Application of the rule-space procedure to language testing: Examining attributes of a free response listening test. Language Testing, 15(2), 119-157.
6. Carr, N. T. (2003). An investigation into the structure of text characteristics and reader abilities in a test of second language reading (Unpublished PhD dissertation). University of California, Department of Applied Linguistics, Los Angeles, USA.
7. Chen, Y. H., Ferron, J. M., Thompson, M. S., Gorin, J. S., & Tatsuoka, K. K. (2010). Group comparisons of mathematics performance from a cognitive diagnostic perspective. Educational Research and Evaluation, 16(4), 325-343.
8. de la Torre, J., & Douglas, J. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353.

9. Embretson, S. (1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93, 179-197.
10. Embretson, S. (1984). A general latent trait model for response processes. Psychometrika, 49, 175-186.
11. Embretson, S. (1991). A multidimensional latent trait model for measuring learning and change. Psychometrika, 37, 359-74.
12. Embretson, S. (1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3, 380-396.
13. Embretson, S., & Gorin, J. (2001). Improving construct validity with cognitive psychology principles. Journal of Educational Measurement, 38(4), 343-368.
14. Freedle, R., & Kostin, I. (1993). The prediction of TOEFL reading comprehension item difficulty for expository prose passages for three item types: Main idea, inference, and supporting idea items. (TOEFL Research Reports No. RR-93-44). Princeton, NJ: Educational Testing Service.
15. Fu, J., & Li, Y. (2007, April). Cognitively diagnostic psychometric models: An integrative review. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.
16. Gao, L., & Rogers, T. (2011). Use of tree-based regression in the analyses of L2 reading test items. Language Testing, 28(1), 77-104.
17. Gierl, M. J. (2007). Attributes using the rule-space model and attribute hierarchy method. Journal of Educational Measurement 44(4), 325-340.
18. Gierl, M. J., & Cui, Y. (2008). Defining characteristics of diagnostic classification models and the problem of retrofitting in cognitive diagnostic assessment. Measurement: Interdisciplinary Research & Perspective, 6(4), 263-268.
19. Gierl, M. J., Cui, Y., & Zhou, J. (2009). Reliability and attribute-based scoring in cognitive diagnostic assessment. Journal of Educational Measurement, 46(3), 293-313.
20. Gierl, M. J., Leighton, J. P., & Hunka, S. (2000). Exploring the logic of Tatsuoka’s rule space model for test development and analysis. Educational Measurement: Issues and Practice, 19, 34-44.
21. Gomez, P. G., Noah, A., Schedl, M., Wright, C., & Yolkut, A. (2007). Proficiency descriptors based on a scale-anchoring study of the new TOEFL iBT reading test. Language Testing, 24(3), 417-444.
22. Gorin, J. S. (2007). Test construction and diagnostic testing. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 173-201). New York: Cambridge University Press.
23. Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333-352.
24. Hartz, S. M. (2002). A Bayesian guide for the unified model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation). University of Illinois, Department of Statistics, Urbana-Champaign, IL.
25. Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29, 262-277.
26. Henson, R., Templin, J., & Willse, J. (2009). Defining a family of cognitive diagnosis models using log linear models with latent variables. Psychometrika, 74, 191-210.
27. Huff, K., & Goodman, D. P. (2007). The demand for cognitive diagnostic assessment. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 19-60). New York: Cambridge University Press.
28. Im, S., & Park, H. J. (2010). A comparison of US and Korean students' mathematics skills using a cognitive diagnostic testing method: linkage to instruction. Educational Research and Evaluation, 16(3), 287-301.
29. Jang, E. E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL (Unpublished doctoral dissertation). University of Illinois at Urbana- Champaign.
30. Jang, E. E. (2008). A framework for cognitive diagnostic assessment. In C. A. Chapelle, Y. R. Chung, & J. Xu (Eds.), Towards adaptive CALL: Natural language processing for diagnostic language assessment (pp. 117-131). Ames, IA: Iowa State University.
31. Jang, E. E. (2009a). Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for Fusion Model application to LanguEdge assessment. Language Testing, 26(1), 31-73.
32. Jang, E. E. (2009b). Demystifying a Q-matrix for making diagnostic inferences about L2 reading skills. Language Assessment Quarterly, 6(3), 210-238.
33. Junker, B., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.
34. Jurich, D. P., & Bradshaw, L. P. (2014). An illustration of diagnostic classification modeling in student learning outcomes assessment. International Journal of Testing, 14(1), 37-41.
35. Kim, A. (2015). Exploring ways to provide diagnostic feedback with an ESL placement test: Cognitive diagnostic assessment of L2 reading ability. Language Testing, 3(2) 227-258.
36. Kostin, I. (2004). Exploring item characteristics that are related to the difficulty of TOEFL dialogue items (TOEFL Research Rep. No. RR-79). Princeton, NJ: Educational Testing Service.
37. Lee, Y. W., & Sawaki, Y. (2009a). Cognitive diagnosis approaches to language assessment: An overview. Language Assessment Quarterly, 6(3), 172-189.
38. Lee, Y. W., & Sawaki, Y. (2009b). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239-263.
39. Leighton, J. P., & Gierl, M. J. (2007a). Cognitive diagnostic assessment for education: Theory and applications. New York: Cambridge University Press.
40. Leighton, J. P., & Gierl, M. J. (2007b). Defining and evaluating models of cognition used in educational measurement to make inferences about examinees’ thinking processes. Educational Measurement: Issues and Practice, 26(2), 3-16.
41. Li, H., & Suen, H. K. (2013). Constructing and validating a Q-Matrix for cognitive diagnostic analyses of a reading test. Educational Assessment, 18(1), 1-25.
42. Madison, M., & Bradshaw, L. (2014). The effects of Q-matrix design on classification accuracy in the log-linear cognitive diagnosis model. Educational and Psychological Measurement, 75(3), 491-511.
43. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13- 103). New York: Macmillan.
44. Mislevy, R. J. (1996). Test theory reconceived. Journal of Educational Measurement, 33, 379-416.
45. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-62.
46. National Research Council. (2001). Knowing what students know: The science and design of educational assessment. Washington DC: National Academy.
47. Nichols, P. D. (1994). A framework for developing cognitively diagnostic assessments. Review of Educational Research, 64(4), 575-603.
48. Nichols, P. D., Chipman, S. F., & Brennan, R. L. (1995). Cognitively diagnostic assessment. Hillsdale, NJ: Erlbaum.
49. Roussos, L. A., Templin, J. L., & Henson, R. A. (2007). Skills diagnosis using IRT based latent class models. Journal of Educational Measurement, 44, 293-311.
50. Roussos, L., DiBello, L., Stout, W., Hartz, S., Henson, R., & Templin, J. (2007). The fusion model skills diagnostic system. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment in education: Theory and applications (pp. 275-318). New York: Cambridge University Press.
51. Rupp, A. A. (2007): The answer is in the question: A guide for describing and investigating the conceptual foundations and statistical properties of cognitive psychometric models. International Journal of Testing, 7(2), 95-125.
52. Rupp, A. A., & Mislevy, R. J. (2007). Cognitive Foundations of Structured Item Response Models. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment in education: Theory and applications (pp. 205-341). New York: Cambridge University Press.
53. Rupp, A. A., & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement: Interdisciplinary Research and Perspectives, 6(4), 219-262.
54. Rupp, A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York: Guilford.
55. Sawaki, Y., Kim, H. J., & Gentile, C. (2009). Q-matrix construction: defining the link between constructs and test items in large-scale reading and listening comprehension assessments. Language Assessment Quarterly, 6(3), 190-209.
56. Snow, R. E., & Lohman, D. F. (1989). Implication of cognitive psychology for education measurement. In R.L. Linn (Ed.), Educational measurement (pp. 263- 331). New York: Macmillan.
57. Stanovich, K. E. (1980). Towards an interactive compensatory model of individual differences in the development of reading fluency. Reading Research Quarterly, 16, 32-71.
58. Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.
59. Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305.
60. Templin, J., & Bradshaw, L. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30, 251-275.
61. von Davier, M. (2007). Hierarchical general diagnostic models (Research Report No. 07-19). Princeton, NJ: Educational Testing Service.
62. Xu, X., & von Davier, M. (2008). Linking for the general diagnostic model. ETS Research Report. Princeton, New Jersey: ETS.
63. Yang, X., & Embretson, S. (2007). Construct validity and cognitive diagnostic assessment. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 119-145). New York: Cambridge University Press.