A Diagnostic Comparison of Turkish and Korean Students’ Mathematics Performances on the TIMSS 2011 Assessment

Abstract

The purpose of the present study was to analyze an international large-scale data set using a cognitive assessment approach. Although some researchers question the usefulness of international large-scale assessments (e.g., TIMSS), participating countries have continued to use the results from these large-scale assessments to improve their curricula and teaching methods. Despite the common reporting practice—single-score—in these large scale assessments gives useful insights about students’ overall performances, they still lack diagnostic information. Cognitive diagnosis models (CDMs) were developed to provide more feedback on students’ cognitive strengths and weaknesses. This study retrofitted the TIMSS 2011 eighth grade mathematics assessment by applying a specific CDM called the DINA (the deterministic, inputs, noisy, “and” gate) model to data from South Korea and Turkey. Results of the DINA model were used to make a detailed comparison between students of these two countries.

References

1. Choi, M. K., Lee, Y-S., & Park, Y. S. (2015). What CDM can tell about what students have learned: An analysis of TIMSS eighth grade mathematics. Eurasia Journal of Mathematics, Science & Technology Education, 11(6), 1563–1577.
2. Chung, Y.L., & Son, D.H. (2000). Effects of cooperative learning strategy on achievement and science learning attitudes in middle school biology. Journal of the Korean Association for Research in Science Education, 20, 611–623.
3. Common Core State Standards Initiative (2010). The common core state standards for mathematics. Washington, D.C.: Author.
4. Common Core State Standards Initiative (2014). Myths vs. facts. Retrieved from http://www.corestandards.org/about-the-standards/myths-vs-facts/
5. de la Torre, J. (2008). An empirically-based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343–362.
6. de la Torre, J., & Lee, Y. S. (2008, March). Relationships between cognitive diagnosis, CTT and IRT indices: An empirical investigation. Paper presented at the Annual Meeting of the National Council on Measurement in Education, New York.
7. de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115–130.
8. de la Torre, J. (2011). The generalized DINA model framework.

9. Psychometrika, 76(2), 179–199.
10. DiBello, L., Roussos, L. A., & Stout, W. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In C.
11. V. Rao & S. Sinharay (Eds.), Handbook of Statistics (Vol. 26, Psychometrics) (pp. 979–1027). Amsterdam: Elsevier.
12. Dogan, E., & Tatsuoka, K. (2008). An international comparison using a diagnostic testing model: Turkish students’ profile of mathematical skills on TIMSS-R. Educational Studies in Mathematics, 68(3), 263–272.
13. Doornik, J. A. (2003). Object-oriented matrix programming using Ox (version 3.1) [Computer software]. London: Timberlake Consultants Press.
14. Fisher, D. L., & Kim, H. B. (1999, April). Constructivist learning environments in science classes in Korea. Paper presented at the Annual Meeting of the American Educational Research Association,
15. Montreal, Canada.
16. Fu, J., & Li, Y. (2007, April). An integrated review of cognitively diagnostic psychometric models. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
17. Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301–323.
18. Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign, Urbana-Champaign, IL.
19. Henson, R. A., Templin, J. L., & Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210.
20. Holliday, W. G., & Holliday, B. W. (2003). Why using international comparative math and science achievement data from TIMSS is not helpful. The Education Forum, 67, 250–257.
21. House, J.D. (2009). Classroom instructional strategies and science career interest for adolescent students in Korea: Results from the TIMSS 2003 assessment. Journal of Instructional Psychology, 36, 13–19.
22. Im, S., & Park, H. J. (2010). A comparison of US and Korean students' mathematics skills using a cognitive diagnostic testing method: linkage to instruction. Educational Research and Evaluation, 16(3), 287–301.
23. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.
24. Kim, K., Kim, S., Kim, N., Park, S., Kim, J., Park, H., & Jung, S. (2008). Characteristics of achievement trend in Korea’s middle and high school students from International Achievement Assessment (TIMSS/PISA) (KICE Research report, RRE-2008-3-1). Seoul, Korea: Korea Institute of Curriculum and Evaluation.
25. Lee, Y.-S., Park, Y.S., & Taylan, D. (2011). A cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the U.S. national sample using the TIMSS 2007. International Journal of Testing, 11, 144–177.
26. Lee, Y.-S., Johnson, M., Park, J. Y., Sachdeva, R., Zhang, J., & Waldman, M. (2013, April). A multidimensional scaling (mds) approach for investigating students' cognitive weakness and strength on the TIMSS 2007 mathematics assessment. Paper presented at the 2013 Annual Meeting of the American Educational Research Association Conference in San Francisco, CA.
27. Leighton, J., & Gierl, M. (2007). Why cognitive diagnostic assessment? In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education (pp. 3–18). Cambridge: Cambridge University Press.
28. Mullis, I.V.S., Martin, M.O., Foy, P., & Arora, A. (2012). TIMSS 2011 International Results in Mathematics. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
29. Nichols, P. D., Chipman, S. F., & Brennan, R. L. (Eds.). (2012). Cognitively diagnostic assessment. Hillsdale, NJ: Erlbaum.
30. Oh, S. (2005). Discursive roles of the teacher during class sessions for students presenting their science investigation. International Journal of Science Education, 27, 1825–1851.
31. Ozturk, D., & Ucar, S. (2010). By using TIMSS data, determination and comparison of the factors that affects science achievement of 8 grade students from Taiwan and Turkey. Cukurova University Journal of Social Sciences, 19(3), 241–256.
32. Park, K.M. (2004, July). Mathematics teacher education in East Asian countries: From the perspective of pedagogical content knowledge. Paper presented at the 10th International Congress on Mathematical Education, Copenhagen, Denmark.
33. Roussos, L., DiBello, L. V., Stout, W., Hartz, S., Henson, R. A., & Templin, J. H. (2007). The fusion model skills diagnosis system. In J. P. Leighton, & Gierl, M. J. (Ed.), Cognitively diagnostic assessment for education: Theory and practice. (pp. 275–318). Thousand Oaks, CA: SAGE.
34. Rupp, A. A., & Templin, J. L. (2008a). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement, 6(4), 219–262.
35. Rupp, A. A., & Templin, J. (2008b). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78–96.
36. Rupp, A. A., Templin, J. L., & Henson, R. A. (2010). Diagnostic assessment: Theory, methods, and applications. New York: Guilford Press.
37. Tatsuoka, K. (1985). A probabilistic model for diagnosing misconceptions in the pattern classification approach. Journal of Educational Statistics, 12, 55–73.
38. Templin, J., & Henson, R. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305.
39. Toker, T., & Green, K. (2012, April). An application of cognitive diagnostic assessment on TIMMS-2007 8th Grade Mathematics items. Paper presented at the annual meeting of the American Educational Research Association, Vancouver, British Columbia, Canada.
40. von Davier, M. (2005). A general diagnostic model applied to language testing data. ETS Research Report RR-05-16.
41. Wang, J. (2001). TIMSS primary and middle school data: Some technical concerns. Educational Researcher, 30, 17–21.