Item parameter recovery via traditional 2PL, Testlet and Bi-factor models for Testlet-Based tests

Abstract

The testlet comprises a set of items based on a common stimulus. When the testlet is used in the tests, there may violate the local independence assumption, and in this case, it would not be appropriate to use traditional item response theory models in the tests in which the testlet is included. When the testlet is discussed, one of the most frequently used models is the testlet response theory (TRT) model. In addition, the bi-factor model and traditional 2PL models are also used for testlet-based tests. This study aims to examine the item parameters estimated by these three calibration models of the data properties produced under different conditions and to compare the performances of the models. For this purpose, data were generated under three conditions: sample size (500, 1000, and 2000), testlet variance (.25, .50, and 1), and testlet size (4 and 10). For each simulation condition, the number of items in the test was fixed at i = 40 and 100 replications were made under each condition. Among these models, it was concluded that the TRT model gave less biased results than the other two models, but the results of the bi-factor model and the TRT were more similar as the sample size increased. Among the examined conditions, it was determined that the most effective variable in parameter recovery was the sample size.

Keywords

• Local dependence
• Testlet model
• Bi-factor model
• Random effect parameter
• Parameter recovery

References

1. Bradlow, E.T., Wainer, H., & Wang, X. (1999). A Bayesian random effects model for testlets. Psychometrika, 64(2), 153-168. https://doi.org/10.1007/bf02294533
2. Chalmers, R.P. (2020). mirt: Multidimensional item response theory. R package version 1.33.2. [Computer software manual]. http://www.R-project.org/
3. Chen, W.H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22(3), 265–289. https://doi.org/10.3102/10769986022003265
4. DeMars, C.E. (2006). Application of the bi‐factor multidimensional item response theory model to testlet based tests. Journal of Educational Measurement, 43(2), 145 168. https://doi.org/10.1111/j.1745-3984.2006.00010.x
5. DeMars, C.E. (2012). Confirming testlet effects. Applied Psychological Measurement, 36, 104 121. https://doi.org/10.1177/0146621612437403
6. Eckes, T. (2014). Examining testlet effects in the TestDaF listening section: A testlet response theory modeling approach. Language Testing, 31(1), 39 61. https://doi.org/10.1177/0265532213492969
7. Eckes, T., & Baghaei, P. (2015). Using testlet response theory to examine local dependence in C tests. Applied Measurement in Education, 28(2), 85 98. https://doi.org/10.1080/08957347.2014.1002919
8. Glas, C.A.W., Wainer, H., & Bradlow, E.T. (2000). MML and EAP estimation in testlet-based adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 271–288). Kluwer-Nijhoff.

9. Jiao, H., Kamata, A., Wang, S., & Jin, Y. (2012). A multilevel testlet model for dual local dependence. Journal of Educational Measurement, 49(1), 82 100. https://doi.org/10.1111/j.1745-3984.2011.00161.x
10. Jiao, H., Wang, S., & Kamata, A. (2005). Modeling local item dependence with the hierarchical generalized linear model. Journal of Applied Measurement, 6(3), 311-321.
11. Keller, L., Swaminathan, H., & Sireci, S.G. (2003). Evaluating scoring procedures for context-dependent item sets. Applied Measurement in Education, 16, 207 222. https://doi.org/10.1207/s15324818ame1603_3
12. Koziol, N.A. (2016). Parameter recovery and classification accuracy under conditions of testlet dependency: a comparison of the traditional 2PL, testlet, and bi-factor models. Applied Measurement in Education, 29(3), 184 195. https://doi.org/10.1080/08957347.2016.1171767
13. Li, F. (2017). An information‐correction method for testlet‐based test analysis: From the perspectives of item response theory and generalizability theory (Report No. ETS RR-17-27). ETS Research Report Series. https://doi.org/10.1002/ets2.12151
14. Liu Y, & Liu H.Y. (2012). When should we use testlet model? A comparison study of Bayesian testlet random-effects model and standard 2-pl bayesian model. Acta Psychologica Sinica, 44(2), 263-275. https://doi.org/10.3724/sp.j.1041.2012.00263
15. Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Addison-Wesley.
16. Luo, Y., & Wolf, M.G. (2019). Item parameter recovery for the two-parameter testlet model with different estimation methods. Psychological Test and Assessment Modeling, 61(1), 65-89.
17. Min, S., & He, L. (2014). Applying unidimensional and multidimensional item response theory models in testlet-based reading assessment. Language Testing, 31(4), 453-477. https://doi.org/10.1177/0265532214527277
18. Paek, I., & Cole, K. (2019). Using R for item response theory model applications. Routledge.
19. Pak, S. (2017). Ability parameter recovery of a computerized adaptive test based on rasch testlet models [Doctoral dissertation, University of Iowa]. Iowa University Libraries https://doi.org/10.17077/etd.5akqn3gy
20. Rijmen, F. (2010). Formal relations and an empirical comparison among the bi-factor, the testlet, and a second-order multidimensional IRT model. Journal of Educational Measurement, 47, 361-372. https://doi.org/10.1111/j.1745-3984.2010.00118.x
21. Sireci, S.G., Thissen, D., & Wainer, H. (1991). On the reliability of testlet‐based tests. Journal of Educational Measurement, 28(3), 237 247. https://doi.org/10.1002/j.2333 8504.1991.tb01389.x
22. Tao, W., & Cao, Y. (2016). An extension of IRT-based equating to the dichotomous testlet response theory model. Applied Measurement in Education, 29(2), 108 121. https://doi.org/10.1080/08957347.2016.1138956
23. Thissen, D., Steinberg, L., & Mooney, J.A. (1989). Trace lines for testlets: A use of multiplecategorical-response models. Journal of Educational Measurement, 26, 247-260. https://doi.org/10.1111/j.1745-3984.1989.tb00331.x
24. Tuerlinckx, F., & De Boeck, P. (2001). The effect of ignoring item iterations on the estimated discrimination parameters in item response theory. Psychological Methods, 6(2), 181-195. https://doi.org/10.1037/1082-989x.6.2.181
25. Wainer, H. (1995). Precision and differential item functioning on a testlet-based test: The 1991 Law School Admissions Test as an example. Applied Measurement in Education, 8, 157–186. https://doi.org/10.1207/s15324818ame0802_4
26. Wainer, H., Bradlow, E.T., & Du, Z. (2000). Testlet response theory: An analog for the 3PL model useful in testlet-based adaptive testing. In W. J. van der Linden & C.A.W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 245–270). Kluwer-Nijhoff.
27. Wainer, H., Bradlow, E.T., & Wang, X. (2007). Testlet response theory and its applications. Cambridge University Press.
28. Wainer, H., & Kiely, G.L. (1987). Item clusters and computerized adaptive testing: A case for testlets. Journal of Educational Measurement, 24, 185 201. https://doi.org/10.1111/j.1745-3984.1987.tb00274.x
29. Wainer, H., & Thissen, D. (1996). How is reliability related to the quality of test scores? What is the effect of local dependence on reliability? Educational Measurement: Issues and Practice, 15(1), 22-29. https://doi.org/10.1002/j.2333-8504.1998.tb01749.x
30. Wainer, H., & Wang, X. (2000). Using a new statistical model for testlets to score TOEFL. Journal of Educational Measurement, 37(3), 203-220. https://doi.org/10.1002/j.2333-8504.2001.tb01851.x
31. Yen, W.M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30(3), 187 213. https://doi.org/10.1111/j.1745-3984.1993.tb00423.x
32. Zenisky, A.L., Hambleton, R.K., & Sired, S.G. (2002). Identification and evaluation of local item dependencies in the Medical College Admissions Test. Journal of Educational Measurement, 39(4), 291-309. https://doi.org/10.1111/j.1745-3984.2002.tb01144.x