Simultaneous Estimation of Overall Score and Subscores Using MIRT, HO-IRT and Bi-factor Model on TIMSS Data

Abstract

In educational testing, there is an increasing interest in the simultaneous estimation of the overall scores and subscores. This study aims to compare the reliability and precision of the simultaneous estimation of overall scores and sub-scores using MIRT, HO-IRT and Bi-factor models. TIMSS 2015 mathematics scores have been used as a data set in this study. The TIMSS 2015 mathematics test consists of 35 items, four of which are polytomously scored (0-1-2), and the rest of the items are dichotomously scored (0-1). The four content domains include number (14 items), algebra (9 items), geometry (6 items), and data and change (6 items). Ability parameters were estimated using the BMIRT software. The results showed that the MIRT and HO-IRT methods performed similarly in terms of precision and reliability for subscore estimates. The MIRT maximum information method had the smallest standard error of measurement for the overall score estimates. All three methods performed similarly in terms of the overall score reliability. The findings suggest that among the three methods compared, HO-IRT appears to be a better choice in the simultaneous estimation of the overall score and subscores for the data from TIMSS 2015. Recommendations for the testing practices and future research are provided.

Keywords

• TIMSS,subscores,multidimensional item response theory,higher-order item response theory,bi-factor model

References

1. Ackerman, T. A., Gierl, M. J., & Walker, C. M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22, 37-51. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
2. Adams, R., Wilson, M., & Wang, W.-C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1–23. https://doi.org/10.1177/0146621697211001
3. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for educational and psychological testing. Washington, DC: AERA.
4. Brown, A. & Croudace, T. (2015). Scoring and estimating score precision using multidimensional IRT. In Reise, S. P. & Revicki, D. A. (Eds.). Handbook of Item Response Theory Modeling: Applications to Typical Performance Assessment (a volume in the Multivariate Applications Series). New York: Routledge/Taylor & Francis Group.
5. Cai, L., Yang, J. S., & Hansen, M. (2011). Generalized full-information item bifactor analysis. Psychological Methods, 16(3), 221–248. http://doi.org/10.1037/a0023350
6. de la Torre, J., & Patz, R. J. (2005). Making the most of what we have : A practical application of multidimensional item response theory in test scoring. Journal of Educational and Behavioral Statistics, 30(3), 295–311. https://doi.org/10.3102/10769986030003295
7. de la Torre, J., & Hong, Y. (2010). Parameter estimation with small sample size: A higher-order IRT model approach. Applied Psychological Measurement, 34, 267-285. https://doi.org/10.1177/0146621608329501
8. de la Torre, J., & Song, H. (2009). Simultaneous estimation of overall and domain abilities: A higher-order ırt model approach. Applied Psychological Measurement, 33(8), 620–639. http://doi.org/10.1177/0146621608326423

9. de la Torre, J., Song, H., & Hong, Y. (2011). A comparison of four methods of IRT subscoring. Applied Psychological Measurement, 35(4), 296–316. http://doi.org/10.1177/0146621610378653
10. DeMars, C.E. (2005, August). Scoring subscales using multidimensional item response theory models. Poster presented at the annual meeting of the American Psychology Association. Retrieved from https://eric.ed.gov/?id=ED496242
11. DeMars, C. E. (2013). A tutorial on interpreting bifactor model scores. International Journal of Testing, 13, 354-378. https://doi.org/10.1080/15305058.2013.799067
12. Fan, F. (2016). Subscore Reliability and Classification Consistency : A Comparison of Five Methods (Doctoral dissertation, University of Massachusetts Amherst). Retrieved from https://scholarworks.umass.edu/dissertations_2/857/
13. Gibbons, R. D., & Hedeker, D. R. (1992). Full-information item bi-factor analysis. Psychometrika, 57(3), 423–436. http://doi.org/10.1007/BF02295430
14. Haberman, S. J. (2008). When can subscale scores have value? Journal of Educational and Behavioral Statistics, 33, 204–229. https://doi.org/10.3102/1076998607302636
15. Haberman, S. J., & Sinharay, S. (2010). Reporting of subscores using multidimensional item response theory. Psychometrika, 75, 209– 227. https://doi.org/10.1007/s11336-010-9158-4
16. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structural analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1- 55. https://doi.org/10.1080/10705519909540118
17. Jang, E. E., & Roussos, L. (2007). An investigation into the dimensionality of TOEFL using conditional covariance-based nonparametric approach. Journal of Educational Measurement, 44, 1-21. https://doi.org/10.1111/j.1745-3984.2007.00024.x
18. Lane, S. (2005, April). Status and future directions for performance assessments in education. Paper presented at the annual meeting of the American Educational Research Association, Montreal.
19. Liu, Y., Li, Z., & Liu, H. (2018). Reporting Valid and Reliable Overall Scores and Domain Scores Using Bi-Factor Model. Applied Psychological Measurement, 43(7), 562–576. https://doi.org/10.1177/0146621618813093
20. Liu, Y., & Liu, H. (2017). Reporting overall scores and domain scores of bi-factor models. Acta Psychologica Sinica, 49(9), 1234. http://doi.org/10.3724/SP.J.1041.2017.01234
21. Longabach, T. (2015). A comparison of subscore reporting methods for a state assessment of English language proficiency (Doctoral dissertation, University of Kansas). Retrieved from https://kuscholarworks.ku.edu/handle/1808/19517
22. Md Desa, Z. N. D. (2012). Bi-factor multidimensional item response theory modeling for subscore estimation, reliability, and classification (Doctoral dissertation, University of Kansas). Retrieved from http://kuscholarworks.ku.edu/dspace/handle/1808/10126
23. Monaghan, W. (2006). The facts about subscale scores (ETS R&D Connections No. 4). Princeton, NJ: Educational Testing Service. Retrieved from https://www.ets.org/Media/Research/pdf/RD_Connections4.pdf
24. Reckase, M. D. (1985). The difficulty of items that measure more than one ability. Applied Psychological Measurement, 9, 401-412. https://doi.org/10.1177/014662168500900409
25. Reckase, M. D. (1997). A linear logistic multidimensional model for dichotomous item response data. In W. J. van der Linden &R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 271–286). NY: Springer.
26. Reckase, M. D. (2009). Multidimensional item response theory (statistics for social and behavioral sciences). New York: Springer.
27. Reckase, M. D., & Xu, J.-R. (2015). The Evidence for a Subscore Structure in a Test of English Language Competency for English Language Learners. Educational and Psychological Measurement, 75(5), 805–825. https://doi.org/10.1177/0013164414554416
28. Robitzsch, A. (2019). Supplementary Item Response Theory Models Version. R-project, Package 'sirt' manual. Retrieved from https://cran.r-project.org/web/packages/sirt/sirt.pdf
29. Sinharay, S. (2010). How often do subscores have added value? Results from operational and simulated data. Journal of Educational Measurement, 47(2), 150–174. https://doi.org/10.1111/j.1745-3984.2010.00106.x
30. Sinharay, S., Haberman, S. J., & Wainer, H. (2011). Do Adjusted Subscores Lack Validity? Don’t Blame the Messenger. Educational and Psychological Measurement, 71(5), 789–797. https://doi.org/10.1177/0013164410391782
31. Soysal, S., & Kelecioğlu, H. (2018). Toplam Test ve Alt Test Puanlarının Kestiriminin Hiyerarşik Madde Tepki Kuramı Modelleri ile Karşılaştırılması. Journal of Measurement and Evaluation in Education and Psychology , 9(2), 178-201. https://doi.org/10.21031/epod.404089
32. Stone, C. A., Ye, F., Zhu, X., & Lane, S. (2010). Providing subscale scores for diagnostic information: A case study when the test is essentially unidimensional. Applied Measurement in Education, 23(1), 63–86. https://doi.org/10.1080/08957340903423651
33. Tabachnick B. G. & Fidel, L. S. (2001). Using multivariate statistics (4th ed.). MA: Allyn & Bacon, Inc.
34. Wang, W.-C., Chen, P.-H., & Cheng, Y.-Y. (2004). Improving measurement precision of test batteries using multidimensional ıtem response models. Psychological Methods, 9(1), 116–136. http://doi.org/10.1037/1082-989X.9.1.116
35. Wedman, J., & Lyren, P.‐E. (2015). Methods for examining the psychometric quality of subscores: A review and application. Practical Assessment, Research and Evaluation, 20 (21). Retrieved from https://pareonline.net/getvn.asp?v=20&n=21
36. Wainer, H., & Thissen, D. (1996). How is reliability related to the quality of test scores? What is the effect of local dependence on reliability? Educational Measurement: Issues and Practices, 15, 22–29. https://doi.org/10.1111/j.1745-3992.1996.tb00803.x
37. Wang, W.-C., Chen, P.-H., & Cheng, Y.-Y. (2004). Improving measurement precision of test batteries using multidimensional item response models. Psychological Methods, 9(1), 116–136. http://doi.org/10.1037/1082-989X.9.1.116
38. Yao, L. (2003). BMIRT: Bayesian multivariate item response theory [computer software]. Monterey, CA: DefenseManpower Data Center. Downloaded from https://www.bmirt.com/6271.html
39. Yao, L. (2010). Reporting valid and reliable overall scores and domain scores. Journal of Educational Measurement, 47(3), 339–360. http://doi.org/10.1111/j.1745-3984.2010.00117
40. Yao, L. (2013). The BMIRT toolkit. Monterey. Retrieved from https://www.bmirt.com/8220.html
41. Yao, L. (2014). Multidimensional item response theory for score reporting. In Y. Cheng, & H.‐H. Chang (Eds.) Advances in modern international testing: Transition from summative to formative assessment. Charlotte, NC: Information Age.
42. Yao, L., & Boughton, K. A. (2007). A Multidimensional item response modeling approach for improving subscale proficiency estimation and classification. Applied Psychological Measurement, 31(2), 83–105. http://doi.org/10.1177/0146621606291559
43. Yao, L., Lewis, D., & Zhang, L. (2008, April). An introduction to the application of BMIRT: Bayesian multivariate item response theory software. Training session presented at the meeting of the National Council on Measurement in Education, New York, NY.
44. Yao, L., & Schwarz R. (2006). Amultidimensional partial credit model with associated item and test statistics: An application to mixed format tests. Applied Psychological Measurement. 30, 469—492. https://doi.org/10.1177/0146621605284537
45. Zhang, J. (2007). Conditional covariance theory and DETECT for polytomous items. Psychometrika, 72, 69-91. https://doi.org/10.1007/s11336-004-1257-7
46. Zhang, J., & Stout, W. (1999a). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64, 129-152. https://doi.org/10.1007/BF02294532
47. Zhang, J., & Stout, W. (1999b). The theoretical DETECT index of dimensionality and its applicationto approximate simple structure. Psychometrika, 64, 213-249. https://doi.org/10.1007/BF02294536
48. Zhang, M. (2016). Exploring dimensionality of scores for mixed-format tests (Doctoral Dissertation, University of Iowa). Retrieved from https://ir.uiowa.edu/etd/2171/