Araştırma Makalesi
PDF Zotero Mendeley EndNote BibTex Kaynak Göster

Mixed Adaptive Multistage Testing: A New Approach

Yıl 2021, Cilt 12, Sayı 4, 358 - 373, 29.12.2021
https://doi.org/10.21031/epod.871014

Öz

Computerized adaptive testing (CAT) and computerized multistage testing (CMT) are two popular versions of adaptive testing with their own strengths and weaknesses. This study proposes and investigates a combination of the two procedures designed to capture these strengths while minimizing the weaknesses by replacing the standard MST routing module with a CAT-based, item-level routing module. A total of 3000 examinees were simulated from a truncated normal distribution with bounds at -3 and 3, and a simulation study was conducted. Simulation results indicate that the new method provides some efficiency improvements over traditional MST when both routing modules are the same size, and when the item-level routing module is larger, the improvements are greater. The study showed that the proposed test administration model could be used to measure student ability, meaning that our new method resulted in lower mean bias, lower RMSE, and higher correlation than traditional MST. An R package built from the code used for this paper is also introduced in the supplementary file. The limitations of the study and recommendations for future research are also presented.

Kaynakça

  • Armstrong, R. D., Jones, D. H., Li, X., & Wu, L. (1996). A study of a network-flow algorithm and a noncorrecting algorithm for test assembly. Applied Psychological Measurement, 20(1), 89-98. doi: 10.1177/014662169602000108
  • Barrada, J. R., Olea, J., Ponsoda, V., & Abad, F. J. (2008). Incorporating randomness in the Fisher information for improving item‐exposure control in CATs. British Journal of Mathematical and Statistical Psychology, 61(2), 493-513. doi: 10.1348/000711007X230937
  • Barton, M. A., & Lord, F. M. (1981). An upper asymptote for the three-parameter logistic item-response model (ETS RR-81-20). Princeton, NJ: Educational Testing Service. doi: 10.1002/j.2333-8504.1981.tb01255.x
  • Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord, & M. R. Novick, (Eds.), Statistical theories of mental test scores (pp. 17-20). Reading, MA: Addison-Wesley.
  • Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431-444. doi: 10.1177/014662168200600405
  • Briethaupt, K., & Hare, D. R. (2007). Automated simultaneous assembly of multistage testlets for a high-stakes licensing examination. Educational and Psychological Measurement, 67(1), 5-20. doi: 10.1177/0013164406288162
  • Chen, S. Y., Ankenmann, R. D., & Chang, H. H. (2000). A comparison of item selection rules at the early stages of computerized adaptive testing. Applied Psychological Measurement, 24(3), 241-255. doi: 10.1177/01466210022031705
  • ILOG. (2006). ILOG CPLEX 10.0 [User’s Manual]. Paris: ILOG SA. Retrieved from https://www.lix.polytechnique.fr/~liberti/teaching/xct/cplex/usrcplex.pdf
  • Kim, H., & Plake, B. S. (1993). Monte Carlo simulation comparison of two-stage testing and computerized adaptive testing. Atlanta, GA: National Council on Measurement in Education.
  • Kim, S., Moses, T., & Yoo, H. (2015). A comparison of IRT proficiency estimation methods under adaptive multistage testing. Journal of Educational Measurement, 52(1), 70-79. doi: 10.1111/jedm.12063
  • Liu, O. L., Bridgeman, B., Gu, L., Xu, J., & Kong, N. (2015). Investigation of response changes in the GRE revised general test. Educational and Psychological Measurement, 75(6), 1002-1020. doi: 10.1177/0013164415573988
  • Luecht, R. M., & Nungester, R. J. (2000). Computer-adaptive sequential testing. In W. J. van der Linden, & C. A. Glas, Computerized adaptive testing: Theory and practice (pp. 117-128). Netherlands: Springer.
  • Luecht, R. M., & Sireci, S. (2012). A review of models for computer-based testing. New York: The College Board. Retrieved from https://files.eric.ed.gov/fulltext/ED562580.pdf
  • Luecht, R. M., Brumfield, T., & Breithaupt, K. (2006). A testlet assembly design for adaptive multistage . Applied Measurement in Education, 19(3), 189-202. doi: 10.1207/s15324818ame1903_2
  • Magis, D., Yan, D., & von Davier, A. (2017). mstR: Procedures to generate patterns under multistage testing. Retrieved from https://CRAN.R-project.org/package=mstR
  • Patsula, L. N. (1999). A comparison of computerized adaptive testing and multistage testing (Unpublished doctoral dissertation). University of Massachusett, Arherst, MA.
  • R Development Core Team . (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.r-project.org Raborn, A. W. (2018). Package ‘caMST’. Retrieved from https://cran.r-project.org/web/packages/caMST/caMST.pdf
  • Reckase, M. D. (2010). Designing item pools to optimize the functioning of a computerized adaptive test. Psychological Test and Assessment Modeling, 52(2), 127-141. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1087.9450&rep=rep1&type=pdf
  • Robin, F., Steffen, M., & Liang, L. (2016). The multistage test implementation of the GRE revised general test. In D. Yan, A. A. von Davier, & C. Lewis (Eds.), Computerized multistage testing (pp. 363-380). Boca Raton, FL: Chapman and Hall/CRC.
  • Sari, H. I., & Huggins-Manley, A. C. (2017). Examining content control in adaptive tests: Computerized adaptive testing vs. computerized adaptive multistage testing. Educational Sciences: Theory & Practice, 17(5), 1759-1781. doi: 10.12738/estp.2017.5.0484
  • Sari, H. I., Yahsi-Sari, H., & Huggins-Manley, A. C. (2016). Computer adaptive multistage testing: Practical issues, challenges and principles. Journal of Measurement and Evaluation in Education and Psychology, 7(2), 388-406. Retrieved from https://dergipark.org.tr/en/download/article-file/270019
  • van der Linden, W. J. (2000). Constrained adaptive testing with shadow tests. In W. J. van der Linden, & G. A. W. Glas (Eds.), Computer adaptive testing: Theory and practice (pp. 27-52). Dordrecht: Kluwer Academic Publishers. doi: 10.1007/0-306-47531-6
  • Wang, S., Lin, H., Chang, H. H., & Douglas, J. (2016). Hybrid computerized adaptive testing: From group sequential design to fully sequential design. Journal of Educational Measurement, 53(1), 45-62. doi: 10.1111/jedm.12100
  • Weiss, D. J., & Kingsbury, G. G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21(4), 361-375. doi: 10.1111/j.1745-3984.1984.tb01040.x Yan, D., von Davier, A. A., & Lewis, C. (2016). Computerized multistage testing: Theory and applications. Boca Raton, FL: CRC Press.
  • Zenisky, A., Hambleton, R. K., & Luecht, R. M. (2010). Multistage testing: Issues, designs, and research. In W. J. van der Linden, & C. A. Glass (Eds.), Elements of adaptive testing (pp. 355-372). New York: Springer.

Yıl 2021, Cilt 12, Sayı 4, 358 - 373, 29.12.2021
https://doi.org/10.21031/epod.871014

Öz

Kaynakça

  • Armstrong, R. D., Jones, D. H., Li, X., & Wu, L. (1996). A study of a network-flow algorithm and a noncorrecting algorithm for test assembly. Applied Psychological Measurement, 20(1), 89-98. doi: 10.1177/014662169602000108
  • Barrada, J. R., Olea, J., Ponsoda, V., & Abad, F. J. (2008). Incorporating randomness in the Fisher information for improving item‐exposure control in CATs. British Journal of Mathematical and Statistical Psychology, 61(2), 493-513. doi: 10.1348/000711007X230937
  • Barton, M. A., & Lord, F. M. (1981). An upper asymptote for the three-parameter logistic item-response model (ETS RR-81-20). Princeton, NJ: Educational Testing Service. doi: 10.1002/j.2333-8504.1981.tb01255.x
  • Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord, & M. R. Novick, (Eds.), Statistical theories of mental test scores (pp. 17-20). Reading, MA: Addison-Wesley.
  • Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431-444. doi: 10.1177/014662168200600405
  • Briethaupt, K., & Hare, D. R. (2007). Automated simultaneous assembly of multistage testlets for a high-stakes licensing examination. Educational and Psychological Measurement, 67(1), 5-20. doi: 10.1177/0013164406288162
  • Chen, S. Y., Ankenmann, R. D., & Chang, H. H. (2000). A comparison of item selection rules at the early stages of computerized adaptive testing. Applied Psychological Measurement, 24(3), 241-255. doi: 10.1177/01466210022031705
  • ILOG. (2006). ILOG CPLEX 10.0 [User’s Manual]. Paris: ILOG SA. Retrieved from https://www.lix.polytechnique.fr/~liberti/teaching/xct/cplex/usrcplex.pdf
  • Kim, H., & Plake, B. S. (1993). Monte Carlo simulation comparison of two-stage testing and computerized adaptive testing. Atlanta, GA: National Council on Measurement in Education.
  • Kim, S., Moses, T., & Yoo, H. (2015). A comparison of IRT proficiency estimation methods under adaptive multistage testing. Journal of Educational Measurement, 52(1), 70-79. doi: 10.1111/jedm.12063
  • Liu, O. L., Bridgeman, B., Gu, L., Xu, J., & Kong, N. (2015). Investigation of response changes in the GRE revised general test. Educational and Psychological Measurement, 75(6), 1002-1020. doi: 10.1177/0013164415573988
  • Luecht, R. M., & Nungester, R. J. (2000). Computer-adaptive sequential testing. In W. J. van der Linden, & C. A. Glas, Computerized adaptive testing: Theory and practice (pp. 117-128). Netherlands: Springer.
  • Luecht, R. M., & Sireci, S. (2012). A review of models for computer-based testing. New York: The College Board. Retrieved from https://files.eric.ed.gov/fulltext/ED562580.pdf
  • Luecht, R. M., Brumfield, T., & Breithaupt, K. (2006). A testlet assembly design for adaptive multistage . Applied Measurement in Education, 19(3), 189-202. doi: 10.1207/s15324818ame1903_2
  • Magis, D., Yan, D., & von Davier, A. (2017). mstR: Procedures to generate patterns under multistage testing. Retrieved from https://CRAN.R-project.org/package=mstR
  • Patsula, L. N. (1999). A comparison of computerized adaptive testing and multistage testing (Unpublished doctoral dissertation). University of Massachusett, Arherst, MA.
  • R Development Core Team . (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.r-project.org Raborn, A. W. (2018). Package ‘caMST’. Retrieved from https://cran.r-project.org/web/packages/caMST/caMST.pdf
  • Reckase, M. D. (2010). Designing item pools to optimize the functioning of a computerized adaptive test. Psychological Test and Assessment Modeling, 52(2), 127-141. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1087.9450&rep=rep1&type=pdf
  • Robin, F., Steffen, M., & Liang, L. (2016). The multistage test implementation of the GRE revised general test. In D. Yan, A. A. von Davier, & C. Lewis (Eds.), Computerized multistage testing (pp. 363-380). Boca Raton, FL: Chapman and Hall/CRC.
  • Sari, H. I., & Huggins-Manley, A. C. (2017). Examining content control in adaptive tests: Computerized adaptive testing vs. computerized adaptive multistage testing. Educational Sciences: Theory & Practice, 17(5), 1759-1781. doi: 10.12738/estp.2017.5.0484
  • Sari, H. I., Yahsi-Sari, H., & Huggins-Manley, A. C. (2016). Computer adaptive multistage testing: Practical issues, challenges and principles. Journal of Measurement and Evaluation in Education and Psychology, 7(2), 388-406. Retrieved from https://dergipark.org.tr/en/download/article-file/270019
  • van der Linden, W. J. (2000). Constrained adaptive testing with shadow tests. In W. J. van der Linden, & G. A. W. Glas (Eds.), Computer adaptive testing: Theory and practice (pp. 27-52). Dordrecht: Kluwer Academic Publishers. doi: 10.1007/0-306-47531-6
  • Wang, S., Lin, H., Chang, H. H., & Douglas, J. (2016). Hybrid computerized adaptive testing: From group sequential design to fully sequential design. Journal of Educational Measurement, 53(1), 45-62. doi: 10.1111/jedm.12100
  • Weiss, D. J., & Kingsbury, G. G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21(4), 361-375. doi: 10.1111/j.1745-3984.1984.tb01040.x Yan, D., von Davier, A. A., & Lewis, C. (2016). Computerized multistage testing: Theory and applications. Boca Raton, FL: CRC Press.
  • Zenisky, A., Hambleton, R. K., & Luecht, R. M. (2010). Multistage testing: Issues, designs, and research. In W. J. van der Linden, & C. A. Glass (Eds.), Elements of adaptive testing (pp. 355-372). New York: Springer.

Ayrıntılar

Birincil Dil İngilizce
Konular Sosyal
Bölüm Makaleler
Yazarlar

Anthony RABORN Bu kişi benim
PASCO COUNTY SCHOOLS
0000-0002-8083-4739
United States


Halil SARI (Sorumlu Yazar)
KİLİS 7 ARALIK ÜNİVERSİTESİ, MUALLİM RIFAT EĞİTİM FAKÜLTESİ
0000-0001-7506-9000
Türkiye

Yayımlanma Tarihi 29 Aralık 2021
Yayınlandığı Sayı Yıl 2021, Cilt 12, Sayı 4

Kaynak Göster

Bibtex @araştırma makalesi { epod871014, journal = {Journal of Measurement and Evaluation in Education and Psychology}, issn = {1309-6575}, eissn = {1309-6575}, address = {}, publisher = {Eğitimde ve Psikolojide Ölçme ve Değerlendirme Derneği}, year = {2021}, volume = {12}, pages = {358 - 373}, doi = {10.21031/epod.871014}, title = {Mixed Adaptive Multistage Testing: A New Approach}, key = {cite}, author = {Raborn, Anthony and Sarı, Halil} }
APA Raborn, A. & Sarı, H. (2021). Mixed Adaptive Multistage Testing: A New Approach . Journal of Measurement and Evaluation in Education and Psychology , 12 (4) , 358-373 . DOI: 10.21031/epod.871014
MLA Raborn, A. , Sarı, H. "Mixed Adaptive Multistage Testing: A New Approach" . Journal of Measurement and Evaluation in Education and Psychology 12 (2021 ): 358-373 <https://dergipark.org.tr/tr/pub/epod/issue/67388/871014>
Chicago Raborn, A. , Sarı, H. "Mixed Adaptive Multistage Testing: A New Approach". Journal of Measurement and Evaluation in Education and Psychology 12 (2021 ): 358-373
RIS TY - JOUR T1 - Mixed Adaptive Multistage Testing: A New Approach AU - Anthony Raborn , Halil Sarı Y1 - 2021 PY - 2021 N1 - doi: 10.21031/epod.871014 DO - 10.21031/epod.871014 T2 - Journal of Measurement and Evaluation in Education and Psychology JF - Journal JO - JOR SP - 358 EP - 373 VL - 12 IS - 4 SN - 1309-6575-1309-6575 M3 - doi: 10.21031/epod.871014 UR - https://doi.org/10.21031/epod.871014 Y2 - 2021 ER -
EndNote %0 Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi Mixed Adaptive Multistage Testing: A New Approach %A Anthony Raborn , Halil Sarı %T Mixed Adaptive Multistage Testing: A New Approach %D 2021 %J Journal of Measurement and Evaluation in Education and Psychology %P 1309-6575-1309-6575 %V 12 %N 4 %R doi: 10.21031/epod.871014 %U 10.21031/epod.871014
ISNAD Raborn, Anthony , Sarı, Halil . "Mixed Adaptive Multistage Testing: A New Approach". Journal of Measurement and Evaluation in Education and Psychology 12 / 4 (Aralık 2021): 358-373 . https://doi.org/10.21031/epod.871014
AMA Raborn A. , Sarı H. Mixed Adaptive Multistage Testing: A New Approach. JMEEP. 2021; 12(4): 358-373.
Vancouver Raborn A. , Sarı H. Mixed Adaptive Multistage Testing: A New Approach. Journal of Measurement and Evaluation in Education and Psychology. 2021; 12(4): 358-373.
IEEE A. Raborn ve H. Sarı , "Mixed Adaptive Multistage Testing: A New Approach", Journal of Measurement and Evaluation in Education and Psychology, c. 12, sayı. 4, ss. 358-373, Ara. 2021, doi:10.21031/epod.871014