Comparison of Testlet Effect on Parameter Estimates Using Different Item Response Theory Models

Abstract

In this study, the testlet effect was calculated for each testlet in the PISA 2018 reading literacy test, and it was examined whether this effect caused a difference in item and ability parameters. The data set was analyzed with a two-parameter logistic item response theory model and a two-parameter logistic testlet model. The results show that variances of testlet effects range from .100 to .432. When the item and ability parameter estimation results of the models were compared, it was determined that the item and ability parameters estimated from the two approaches were highly correlated with each other. It can be said that the item slope and item intercept parameters estimated from different models remained unaffected. However, when the local dependency assumption is not met, it was observed that the standard error values of the two-parameter model for the ability parameter were underestimated. The implications for the analysis and evaluation of the tests based on testlet are discussed. In conclusion, in this study, it was concluded that the testlet effect caused a difference in parameter estimates, but the local dependence among the items was negligible because of the small testlet effects.

Keywords

• Local item dependency
• item response theory
• testlet response theory
• testlet effects
• PISA

References

1. Baghaei, P. & Ravand, H. (2016). Modeling local item dependence in cloze and reading comprehension test items using testlet response theory. Psicológica, 37(1), 85-104.
2. Baker, F. B. (2001). The basics of item response theory. College Park: ERIC Clearinghouse on Assessment and Evaluation, University of Maryland.
3. Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 395–479). Reading, MA: Addison-Wesley.
4. Chang, Y., & Wang, J. (2010). Examining testlet effects on the PIRLS 2006 assessment. Paper presented at 4th IEA International Research Conference, Gothenburg, Sweden. Retrieved from http://www.iea-irc.org/fileadmin/IRC_2010_papers/PIRLS/Chang_Wang.pdf
5. Chalmers, P., Pritikin, J., Robitzsch, A., & Zoltak, M. (2015). Package ‘mirt’. Retrieved January 10, 2021, from https://mran.microsoft.com/snapshot/2014-12-27/web/packages/mirt/mirt.pdf
6. Chen, W. H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22(3), 265–289. https://doi.org/10.3102/10769986022003265
7. DeMars, C. E. (2006). Application of the bi-factor multidimensional item response theory model to testlet-based tests. Journal of Educational Measurement, 43(2), 145-168.
8. Eckes, T. (2014). Examining testlet effects in the TestDaF listening section: A testlet response theory modeling approach. Language Testing, 31(1), 39-61.

9. Eckes, T. & Baghaei, P. (2015). Using testlet response theory to examine local dependence in C-tests. Applied Measurement in Education, 28(2), 85-98.
10. Geramipour, M. (2021). Rasch testlet model and bifactor analysis: how do they assess the dimensionality of large-scale Iranian EFL reading comprehension tests?. Language Testing in Asia, 11(1), 1-23. https://doi.org/10.1186/s40468-021-00118-5
11. Glas, C. A. W., Wainer, H., & Bradlow, E. T. (2000). MML and EAP estimation in testlet-based adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 271-287). Boston, MA: Kluwer-Nijhoff.
12. Ip, E. H. (2010). Interpretation of the three-parameter testlet response model and information function. Applied Psychological Measurement, 34(7), 467-482.
13. Li, F. (2017). An information‐correction method for testlet‐based test analysis: From the perspectives of item response theory and generalizability theory. ETS Research Report Series, (1), 1-25. https://doi.org/10.1002/ets2.12151
14. Li, Y., Bolt, D. M., & Fu, J. (2006). A comparison of alternative models for testlets. Applied Psychological Measurement, 30(1), 3-21.
15. Li, Y., Li, S., & Wang, L. (2010). Application of a general polytomous testlet model to the reading section of a large-scale English language assessment (ETS RR-10–21). Princeton, NJ: Educational Testing Service.
16. Min, S. & He, L. (2014). Applying unidimensional and multidimensional item response theory models in testlet-based reading assessment. Language Testing, 31(4), 453-477.
17. Organisation for Economic Co-operation and Development (2019). PISA 2018 assessment and analytical framework. Paris: OECD Publishing. https://doi.org/10.1787/b25efab8-en
18. Özdemir, B. (2017). Examining testlet effects in english proficiency test: A Bayesian testlet response theory approach. In I. Koleva & G. Duman (Eds.), Educational Research and Practice, (pp. 425-437). Sofia: ST. Kliment Ohridski University Press.
19. Paap, M. C., & Veldkamp, B. P. (2012). Minimizing the testlet effect: Identifying critical testlet features by means of tree-based regression. Psychometrics in Practice at RCEC, 63. Retrieved January 12, 2021, from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1001.1923&rep=rep1&type=pdf#page=71
20. Paek, I., & Cole, K. (2019). Using R for item response theory model applications. London: Routledge.
21. Reckase, M. D. (2009). Multidimensional item response theory models. In Multidimensional item response theory (pp. 79-112). New York, NY: Springer.
22. Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28(3), 237-247.
23. Wainer, H., Bradlow, E. T., & Du. Z. (2000). Testlet response theory. An analog for the 3PL useful in testlet-based adaptive testing. In W. J. van der Linden & G. A. Glas (Eds.). Computerized adaptive testing: Theory and practice (pp. 245-269). Springer, Dordrecht. https://doi.org/10.1007/0-306-47531-6_13
24. Wainer, H., Bradlow, E. T., & Wang, X. (2007). Testlet response theory and its applications. Cambridge: Cambridge University Press.
25. Wainer, H., & Kiely, G. L. (1987). Item clusters and computerized adaptive testing: A case for testlets. Journal of Educational Measurement, 24, 185–201.
26. Wainer, H., & Wang, X. (2000). Using a new statistical model for testlets to score TOEFL. Journal of Educational Measurement, 37(3), 203-220.
27. Wang, W. C., & Wilson, M. (2005). Exploring local item dependence using a random-effects facet model. Applied Psychological Measurement, 29(4), 296–318.
28. Yamamoto, K., Shin, H. J., & Khorramdel, L. (2018). Multistage adaptive testing design in international large‐scale assessments. Educational Measurement: Issues and Practice, 37(4), 16-27.
29. Yen, W. (1993). Scaling performance assessment: Strategies for managing local item dependence. Journal of Educational Measurement, 30, 187-213.
30. Yen, W. M., & Fitzpatrick, A. R. (2006). Item response theory. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 111–153). Westport, CT: American Council on Education/Praeger.
31. Yılmaz Kogar, E., & Kelecioglu, H. (2017). Examination of different item response theory models on tests composed of testlets. Journal of Education and Learning, 6(4), 113-126.