Investigation of a multistage adaptive test based on test assembly methods

Abstract

In this research, multistage adaptive tests (MST) were compared according to sample size, panel pattern and module length for top-down and bottom-up test assembly methods. Within the scope of the research, data from PISA 2015 were used and simulation studies were conducted according to the parameters estimated from these data. Analysis results for each condition were compared in terms of mean RMSE and bias. According to the results obtained from the MST simulation based on the top-down test assembly method, mean RMSE values reduced when the module length increased and when the panel pattern changed from 1-2 to 1-2-2 and 1-2-3 for MST applied to small and large samples. Within the scope of the research, data from PISA 2015 were used and simulation studies were conducted using the parameters estimated from these data. Analysis results for each condition were compared in terms of mean RMSE and bias.

Keywords

• Adaptive multistage test
• Test assembly
• PISA

References

1. American Institute of Certified Public Accountants. (2019, February 18). CPA exam structure. https://www.aicpa.org/becomeacpa/cpaexam/examinationcontent.html
2. Belov, D.I. (2016). Review of modern methods for automated test assembly and item pool analysis. Law School Admission Council Research Report 16-01 March 2016, LSAC Research Report Series, 23 pages, https://www.lsac.org/docs/default-source/research-(lsac-resources)/rr-16-01.pdf
3. Breithaupt, K., Ariel, A., & Veldkamp, B. (2005). Automated simultaneous assembly for multistage testing. International Journal of Testing, 5(3), 319 330. https://doi.org/10.1207/s15327574ijt05038
4. Breithaupt, K., & Hare, D.R. (2007). Automated simultaneous assembly of multistage testlets for a high-stakes licensing examination. Educational and Psychological Measurement, 67(1), 5-20. https://doi.org/10.1177/0013164406288162
5. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. CBS College Publishing.
6. Dallas, A. (2014). The effects of routing and scoring within a computer adaptive multi-stage framework [Unpublished Doctoral Dissertation]. The University of North Carolina.
7. Davis, L.L., & Dodd, B.G. (2003). Item exposure constraints for testlets in the verbal reasoning section of the MCAT. Applied Psychological Measurement, 27(5), 335 356. https://doi.org/10.1177/0146621603256804
8. Educational Testing Service. (2018, February 18). Computer-delivered GRE general test content and structure. http://www.ets.org/gre/revised%5Cgeneral/about/content/computer/

9. Hambleton, R.K., & Xing, D. (2006). Optimal and nonoptimal computer-based test designs for making pass-fail decisions. Applied Measurement in Education, 19(3), 221-239. https://doi.org/10.1207/s15324818ame1903_4
10. Hembry, I.F. (2014). Operational characteristics of mixed format multistage tests using the 3PL testlet response theory model [Unpublished Doctoral Dissertation]. University of Texas at Austin.
11. Hendrickson, A. (2007). An NCME instructional module on multistage testing. Educational Measurement: Issues and Practice, 26(2), 44 52. https://doi.org/10.1111/j.1745 3992.2007.00093.x
12. Hogan, J., Thornton, N., Diaz-Hoffmann, L., Mohadjer, L., Krenzke, T., Li, J. & Khorramdel, L. (2016, July 5,). US program for the international assessment of adult competencies (PIAAC) 2012/2014: Main study and national supplement technical report (NCES 2016-036REV). U.S. Department of Education. National Center for Education Statistics. https://nces.ed.gov/pubs2016/2016036 rev.pdf
13. Jodoin, M.G., Zenisky, A., & Hambleton, R.K. (2006). Comparison of the psychometric properties of several computer-based test designs for credentialing exams with multiple purposes. Applied Measurement in Education, 19(3), 203 220. http://doi.org/10.1207/s15324818ame1903_3
14. Khorramdel, L., Pokropek, A., & van Rijn, P. (2020). Special Topic: Establishing comparability and measurement invariance in large-scale assessments, part I. Psychological Test and Assessment Modeling, 62(1), 3 10. https://www.psychologie aktuell.com/fileadmin/Redaktion/Journale/ptam-2020-1/01_Khorramdel.pdf
15. Kim, S., Moses, T., & You, H. (2015). A comparison of IRT proficiency estimation methods under adaptive multistage testing. Journal of Educational Measurement, 52(1), 70-79. https://doi.org/10.1111/jedm.12063
16. Kim, J., Chung, H., Dodd, B.G., & Park, R. (2012). Panel design variations in the multistage test using the mixed-format tests. Educational and Psychological Measurement, 72(4), 574-588. https://doi.org/10.1177/0013164411428977
17. Kirsch, I., & Lennon, M.L. (2017). PIAAC: A new design for a new era. Large-scale Assessments in Education, 5, 11. https://doi.org/10.1186/s40536-017-0046-6
18. Lord, F.M. (1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum Associates, Inc.
19. Luecht, R. (2000). Implementing the CAST framework to mass produce high quality computer adaptive and mastery tests. Paper presented at the Annual Meeting of the National Council on Measurement in Education (NCME), New Orleans, LA.
20. Luecht, R.M. (2006). Designing tests for pass-fail decisions using item response theory. In S. Downing & T. Haladyna (Eds.), Handbook of test development, 575-596. Lawrence Erlbaum Associates.
21. Luecht, R., Brumfield, T., & Breithaupt, K. (2006). A testlet assembly design for adaptive multistage tests. Applied Measurement in Education, 19(3), 189 202. https://doi.org/10.1207/s15324818ame1903_2
22. Luecht, R.M., & Nungester, R.J. (1998). Some practical examples of computer-adaptive sequential testing. Journal of Educational Measurement, 35(3), 229 249. https://www.learntechlib.org/p/87698/.
23. Luo, X. (2019). Automated test assembly with mixed-ınteger programming: The effects of modeling approaches and solvers. Journal of Educational Measurement, 57(4), 547-565. https://doi.org/10.1111/jedm.12262
24. Luo, X., & Kim, D. (2018). A top-down approach to designing the computerized adaptive multistage test. Journal of Educational Measurement, 55(2), 243 263. https://doi.org/10.1111/jedm.12174
25. Lynn Chen, L.Y. (2010). An investigation of the optimal test design for multi-stage test using the generalized partial credit model [Unpublished Doctoral Dissertation]. The University of Texas at Austin.
26. OECD (2015). PISA 2015 technical report. http://www.oecd.org/pisa/sitedocument/PISA-2015-Technical-Report-Chapter-1-Programme-for-International-Student-Assessment-an-Overview.pdf
27. OECD (2017). PISA 2015 technical report. http://www.oecd.org/pisa/sitedocument/PISA-2015-Technical-Report-Chapter-9-Scaling-PISA-Data.pdf
28. Papadimitriou, C.H., & Steiglitz, K. (1982). Combinatorial optimization: Algorithms and complexity. Prentice-Hall.
29. Park R. (2015). Investigating the impact of a mixed-format item pool on optimal test designs for multistage testing [Unpublished doctoral dissertation]. University of Texas, Austin.
30. Patsula, L.N. (1999). A comparison of computerized-adaptive testing and multi-stage testing [Unpublished doctoral dissertation]. University of Massachusetts at Amherst.
31. Pihlainen, K.A.I., Santtila, M., Häkkinen, K., & Kyröläinen, H. (2018). Associations of physical fitness and body composition characteristics with simulated military task performance. The Journal of Strength & Conditioning Research, 32(4), 1089-1098. https://doi.org/10.1519/jsc.0000000000001921
32. Sari, H.İ. (2016). Examining content control in adaptive tests: Computerized adaptive testing vs. computerized multistage testing [Unpublished doctoral dissertation]. University of Florida.
33. Sari, H.I., & Raborn, A. (2018). What information works best? A comparison of routing methods. Applied psychological measurement, 42(6), 499 515. https://doi.org/10.1177/0146621617752990
34. Şahin Kürşad, M., Çokluk-bökeoglu, Ö. & Çıkrıkçı, N. (2022). The study of the effect of item parameter drift on ability estimation obtained from adaptive testing under different conditions. International Journal of Assessment Tools in Education, 9(3), 654-681. https://doi.org/10.21449/ijate.1070848
35. Theunissen, T.J.J.M. (1985). Binary programming and test design. Psychometrika, 50(4), 411-420. https://link.springer.com/article/10.1007/BF02296260
36. Tian, C. (2018). Comparison of four stopping rules in computerized adaptive testing and examination of their application to on-the-fly multistage testing [Unpublished master dissertation]. University of Illinois.
37. van der Linden, W.J., & Glas, C.A.W. (2000). Capitalization on item calibration error in adaptive testing. Applied Measurement in Education, 13, 35 53. https://doi.org/10.1207/s15324818ame1301_2
38. van der Linden, W.J. (2005). Linear models of optimal test design. Springer.
39. van der Linden, W.J., & Boekkooi-Timminga, E. (1989). A maximin model for IRT-based test design with practical constraints. Psychometrika, 54(2), 237 247. https://link.springer.com/article/10.1007/BF02294518
40. Veldkamp, B.P. (1999). Multiple-objective test assembly problems. Journal of Educational Measurement, 36, 253-66. http://www.jstor.org/stable/1435157
41. Veldkamp, B.P., Matteucci, M., & de Jong, M.G. (2013). Uncertainties in the item parameter estimates and robust automated test assembly. Applied Psychological Measurement, 37, 123-139. https://doi.org/10.1177/0146621612469825
42. Wang, K. (2017). Fair comparison of the performance of computerized adaptive testing and multistage adaptive testing [Unpublished doctoral dissertation]. Michigan State University.
43. Wise, S.L., & Kingsbury, G.G. (2000). Practical issues in developing and maintaining a computerized adaptive testing program. Psicologica, 21, 135 155. https://www.uv.es/revispsi/articulos1y2.00/wise.pdf
44. Xiao, J., & Bulut, O. (2022). Item selection with collaborative filtering in on-the-fly multistage adaptive testing. Applied Psychological Measurement, 46(8), 690 704. https://doi.org/10.1177/01466216221124089
45. Xing, D., & Hambleton, R.K. (2004). Impact of test design, item quality, and item bank size on the psychometric properties of computer-based credentialing examinations. Educational and Psychological Measurement, 64, 5-21. https://doi.org/10.1177/0013164403258393
46. Xu, L., Wang, S., Cai, Y., & Tu, D. (2021). The automated test assembly and routing rule for multistage adaptive testing with multidimensional item response theory. Journal of Educational Measurement, 58, 538-563. https://doi.org/10.1111/jedm.12305
47. Yan, D., Lewis, C., & von Davier, A. (2014). Overview of computerized multistage tests. In D. Yan, A.A. von Davier, & C. Lewis (Eds.). Computerized Multistage Testing: Theory and Applications, 3-20. Chapman & Hall.
48. Yan, D., von Davier, A.A., & Lewis, C. (Eds.). (2014). Computerized Multistage Testing: Theory and Applications (1st ed.). Chapman and Hall/CRC. https://doi.org/10.1201/b16858
49. Yang, L. (2016). Enhancing item pool utilization when designing multistage computerized adaptive tests [Unpublished doctoral dissertation]. Michigan State University.
50. Zenisky, A. (2004). Evaluating the effects of several multistage testing design variables on selected psychometric outcomes for certification and licensure assessment [Unpublished doctoral dissertation]. University of Massachusetts at Amherst.
51. Zenisky, A., & Hambleton, R. (2014). Multistage test designs: Moving research results into practice. In Yan, D., Von Davier, A., & Lewis, C. (Eds.), Computerized Multistage Testing: Theory and Applications, 21-36. Chapman & Hall.
52. Zenisky, A., Hambleton, R.K. & Luecht, R.M. (2010). Multistage testing: Issues, designs and research. In: der Linden, W.J. & Glas, C.A.W. (Eds.). Elements of Adaptive Testing. 355-372. Springer.
53. Zenisky, A.L., Sireci, S.G., Martone, A., Baldwin, P., & Lam, W. (2009). Massachusetts adult proficiency tests technical manual supplement: 2008-2009. Center for Educational Assessment Research. http://www.umass.edu/remp/docs/MAPTTMSupp7-09 final.pdf
54. Zheng, Y. (2014). New methods of online calibration for item bank replenishment [Unpublished Doctoral Dissertation]. University of Illinois at Urbana-Champaign.
55. Zheng, Y., & Chang, H.-H. (2015). On-the-Fly assembled multistage adaptive testing. Applied Psychological Measurement, 39(2), 104 118. https://doi.org/10.1177/0146621614544519
56. Zheng, Y., Nozawa, Y., Gao, X., & Chang, H. (2012). Multistage adaptive testing for a large-scale classification test: the designs, heuristic assembly, and comparison with other testing modes. Paper presented at the Annual Meeting of the National Council on Measurement in Education (NCME) (ACT Research Reports 2012-6). Vancouver, British Columbia, Canada.
57. Zheng, Y., Nozawa, Y., Zhu, R., & Gao, X. (2016). Automated top-down heuristic assembly of a classification multistage test. Int. J. Quantitative Research in Education, 3(4), 242-265. https://doi.org/10.1504/IJQRE.2016.082387
58. Zimowski, M.F., Muraki, E., Mislevy, R.J., & Bock, R.D. (1996). BILOG-MG: Multiple-group IRT analysis and test maintenance for binary items. [Computer software]. Scientific Software International.