Mixed Adaptive Multistage Testing: A New Approach

Abstract

Computerized adaptive testing (CAT) and computerized multistage testing (CMT) are two popular versions of adaptive testing with their own strengths and weaknesses. This study proposes and investigates a combination of the two procedures designed to capture these strengths while minimizing the weaknesses by replacing the standard MST routing module with a CAT-based, item-level routing module. A total of 3000 examinees were simulated from a truncated normal distribution with bounds at -3 and 3, and a simulation study was conducted. Simulation results indicate that the new method provides some efficiency improvements over traditional MST when both routing modules are the same size, and when the item-level routing module is larger, the improvements are greater. The study showed that the proposed test administration model could be used to measure student ability, meaning that our new method resulted in lower mean bias, lower RMSE, and higher correlation than traditional MST. An R package built from the code used for this paper is also introduced in the supplementary file. The limitations of the study and recommendations for future research are also presented.

Keywords

• Computerized adaptive test
• multistage adaptive test
• simulation
• R
• mixed adaptive test

References

1. Armstrong, R. D., Jones, D. H., Li, X., & Wu, L. (1996). A study of a network-flow algorithm and a noncorrecting algorithm for test assembly. Applied Psychological Measurement, 20(1), 89-98. doi: 10.1177/014662169602000108
2. Barrada, J. R., Olea, J., Ponsoda, V., & Abad, F. J. (2008). Incorporating randomness in the Fisher information for improving item‐exposure control in CATs. British Journal of Mathematical and Statistical Psychology, 61(2), 493-513. doi: 10.1348/000711007X230937
3. Barton, M. A., & Lord, F. M. (1981). An upper asymptote for the three-parameter logistic item-response model (ETS RR-81-20). Princeton, NJ: Educational Testing Service. doi: 10.1002/j.2333-8504.1981.tb01255.x
4. Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord, & M. R. Novick, (Eds.), Statistical theories of mental test scores (pp. 17-20). Reading, MA: Addison-Wesley.
5. Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431-444. doi: 10.1177/014662168200600405
6. Briethaupt, K., & Hare, D. R. (2007). Automated simultaneous assembly of multistage testlets for a high-stakes licensing examination. Educational and Psychological Measurement, 67(1), 5-20. doi: 10.1177/0013164406288162
7. Chen, S. Y., Ankenmann, R. D., & Chang, H. H. (2000). A comparison of item selection rules at the early stages of computerized adaptive testing. Applied Psychological Measurement, 24(3), 241-255. doi: 10.1177/01466210022031705
8. ILOG. (2006). ILOG CPLEX 10.0 [User’s Manual]. Paris: ILOG SA. Retrieved from https://www.lix.polytechnique.fr/~liberti/teaching/xct/cplex/usrcplex.pdf

9. Kim, H., & Plake, B. S. (1993). Monte Carlo simulation comparison of two-stage testing and computerized adaptive testing. Atlanta, GA: National Council on Measurement in Education.
10. Kim, S., Moses, T., & Yoo, H. (2015). A comparison of IRT proficiency estimation methods under adaptive multistage testing. Journal of Educational Measurement, 52(1), 70-79. doi: 10.1111/jedm.12063
11. Liu, O. L., Bridgeman, B., Gu, L., Xu, J., & Kong, N. (2015). Investigation of response changes in the GRE revised general test. Educational and Psychological Measurement, 75(6), 1002-1020. doi: 10.1177/0013164415573988
12. Luecht, R. M., & Nungester, R. J. (2000). Computer-adaptive sequential testing. In W. J. van der Linden, & C. A. Glas, Computerized adaptive testing: Theory and practice (pp. 117-128). Netherlands: Springer.
13. Luecht, R. M., & Sireci, S. (2012). A review of models for computer-based testing. New York: The College Board. Retrieved from https://files.eric.ed.gov/fulltext/ED562580.pdf
14. Luecht, R. M., Brumfield, T., & Breithaupt, K. (2006). A testlet assembly design for adaptive multistage . Applied Measurement in Education, 19(3), 189-202. doi: 10.1207/s15324818ame1903_2
15. Magis, D., Yan, D., & von Davier, A. (2017). mstR: Procedures to generate patterns under multistage testing. Retrieved from https://CRAN.R-project.org/package=mstR
16. Patsula, L. N. (1999). A comparison of computerized adaptive testing and multistage testing (Unpublished doctoral dissertation). University of Massachusett, Arherst, MA.
17. R Development Core Team . (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.r-project.org Raborn, A. W. (2018). Package ‘caMST’. Retrieved from https://cran.r-project.org/web/packages/caMST/caMST.pdf
18. Reckase, M. D. (2010). Designing item pools to optimize the functioning of a computerized adaptive test. Psychological Test and Assessment Modeling, 52(2), 127-141. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1087.9450&rep=rep1&type=pdf
19. Robin, F., Steffen, M., & Liang, L. (2016). The multistage test implementation of the GRE revised general test. In D. Yan, A. A. von Davier, & C. Lewis (Eds.), Computerized multistage testing (pp. 363-380). Boca Raton, FL: Chapman and Hall/CRC.
20. Sari, H. I., & Huggins-Manley, A. C. (2017). Examining content control in adaptive tests: Computerized adaptive testing vs. computerized adaptive multistage testing. Educational Sciences: Theory & Practice, 17(5), 1759-1781. doi: 10.12738/estp.2017.5.0484
21. Sari, H. I., Yahsi-Sari, H., & Huggins-Manley, A. C. (2016). Computer adaptive multistage testing: Practical issues, challenges and principles. Journal of Measurement and Evaluation in Education and Psychology, 7(2), 388-406. Retrieved from https://dergipark.org.tr/en/download/article-file/270019
22. van der Linden, W. J. (2000). Constrained adaptive testing with shadow tests. In W. J. van der Linden, & G. A. W. Glas (Eds.), Computer adaptive testing: Theory and practice (pp. 27-52). Dordrecht: Kluwer Academic Publishers. doi: 10.1007/0-306-47531-6
23. Wang, S., Lin, H., Chang, H. H., & Douglas, J. (2016). Hybrid computerized adaptive testing: From group sequential design to fully sequential design. Journal of Educational Measurement, 53(1), 45-62. doi: 10.1111/jedm.12100
24. Weiss, D. J., & Kingsbury, G. G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21(4), 361-375. doi: 10.1111/j.1745-3984.1984.tb01040.x Yan, D., von Davier, A. A., & Lewis, C. (2016). Computerized multistage testing: Theory and applications. Boca Raton, FL: CRC Press.
25. Zenisky, A., Hambleton, R. K., & Luecht, R. M. (2010). Multistage testing: Issues, designs, and research. In W. J. van der Linden, & C. A. Glass (Eds.), Elements of adaptive testing (pp. 355-372). New York: Springer.