Effect of Routing Methods on the Performance of Multi-Stage Tests

Year 2022, , 343 - 354, 30.10.2022

Başak Erdem Kara

https://doi.org/10.46778/goputeb.1123902

Cited By: 1

Abstract

In recent decades, thanks to the great advances and growing opportunities in the technology world, computer-based testing has become a popular alternative to the traditional fixed-item paper-pencil tests. Specifically, multi-stage tests (MST) which is a kind of CBT and an algorithm-based approach become a viable alternative to traditional fixed-item tests with important measurement advantages they provided. This study aimed to examine the effect of different routing rules and scoring methods under different ability distributions in MSTs. For this purpose, three different routing rule, three different ability estimation methods, and two different ability distributions were manipulated in a simulation design. Although there was no clear best method in the studied conditions, it was seen that the Kullback-Leibler was the most efficient routing method and worked best with the EAP scoring method in most of the conditions. Furthermore, EAP and BM provided higher measurement efficiency than the ML method. Recommendations for using those routing methods were provided and suggestions were made for further research.

Keywords

Multi-Stage Testing, Routing, Ability Estimation

Yönlendirme Yöntemlerinin Çok Aşamalı Testler Üzerindeki Etkisi

Abstract

Özellikle son yıllarda, teknoloji dünyasındaki gelişmeler ve artan olanaklarla birlikte, bilgisayara dayalı testlerin popülerliği artmış ve bu testler geleneksel kâğıt-kalem testlerin yerine uygulanabilir bir alternatif halini almıştır. Bilgisayara dayalı testlerin bir türü olan ve algoritmaya dayalı bir yaklaşım olan Çok Aşamalı Testler de sağladıkları önemli avantajlarla birlikte kağıt-kalem testlerinin önemli bir alternatifi haline gelmiştir. Bu çalışmanın amacı, Çok Aşamalı Testlerde yönlendirme yöntemlerinin test performansına etkisinin farklı koşullar altında incelenmesidir. Bu amaçla bir simülasyon çalışması tasarlanmış, üç farklı yönlendirme kuralı, üç farklı yetenek kestirim yöntemi ve iki farklı yetenek dağılımı manipüle edilmiştir. Analizler sonucunda hem normal hem de uniform dağılım için, birçok koşulda Kullback-Leibler'in en etkili yönlendirme yöntemi olduğu ve koşulların çoğunda EAP puanlama yöntemiyle en iyi şekilde çalıştığı görülmüştür. Ayrıca, EAP ve BM yetenek kestirim yöntemleri, ML yönteminden daha yüksek ölçüm hassasiyeti sağlamıştır. En düşük ölçüm hassasiyeti ise, tesadüfi yönlendirme yönteminde elde edilmiştir. Bu yönlendirme yöntemlerinin kullanımına ve ileriki araştırmalara yönelik bazı önerilerde bulunulmuştur.

Keywords

Çok Aşamalı Testler, Yönlendirme, Yetenek Kestirimi

References

• Diao, Q., & Reckase, M. (2009). Comparison of ability estimation and item selection methods in multidimensional computerized adaptive testing. In D. J. Weiss (Ed.), Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing. Retrieved [13.03.2021] from https://www.psych.umn.edu/psylabs/CATCentral
• Haberman, S. J. & von Davier, A. A. (2014). Considerations on parameter estimation, scoring, and linking in multistage testing. In D. Yan, A. A. von-Davier & C. Lewis (Eds.), Computerized multistage testing (p. 229 – 246). CRC Press; Taylor&Francis Group.
• Harwell, M., Stone, C. A., Hsu, T., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20(2), 101–125.
• Hendrickson, A. (2007). An NCME instructional module on multistage testing. Educational Measurement: Issues and Practice, Summer 2007, 44-52.
• Kim, S., Moses, T., & Yoo, H. (2015). A comparison of IRT proficiency estimation methods under adaptive multistage testing. Journal of Educational Measurement, 52(1), 70-79.
• Lord, F. M. (1980). Applications of item response theory to practical testing problems. Mahwah, New Jersey: Routledge
• Magis, D., Yan, D. & von-Davier, A. (Eds.). (2017). Computerized adaptive and multistage testing with R: Using packages catR and mstR. Springer.
• Magis, D., Yan, D. & von Davier, A., A. (2018). Package ‘mstR’: Procedures to generate patterns under multistage testing. Retrieved from https://cran.microsoft.com/snapshot/2018-09-29/web/packages/mstR/mstR.pdf
• OECD (2013). Technical report of the survey of adult skills (PIAAC). OECD Publishing: Paris, France.
• Sarı, H. İ. (2016). Examining content control in adaptive tests: Computerized adaptive testing vs. computerized multistage testing. Unpublished doctoral dissertation. University of Florida, USA.
• Sarı, H. İ., & Raborn, A. (2018). What information works best?: A comparison of routing methods. Applied Psychological Measurement, 42(6), 499–515. DOI: 10.1177/0146621617752990
• Svetina, D., Liaw, Y. L., Rutkowski L. & Rutkowski, D. (2019). Routing strategies and optimizing design for multistage testing in international large-scale assessments. Journal of Educational Measurement, 56(1), 192-213.
• Wainer, H. (2000). Introduction and history. In H. Wainer (Ed.), Computerized adaptive testing: A primer (2nd Edition), (p. 1–22). Lawrence Erlbaum Associates.
• Wang, S., Lin, H., Chang, H. H., Douglas, J. (2016). Hybrid computerized adaptive testing: From group sequential design to fully sequential design. Journal of Educational Measurement, 53, 45-62.
• Wang, K. (2017). A fair comparison of the performance of computerized adaptive testing and multistage adaptive testing. Unpublished doctoral dissertation. Michigan State University, USA.
• Weissman, A., Belov, D.I., & Armstrong, R.D. (2007). Information-based versus number-correct routing in multistage classification tests (Research Report RR-07–05). Law School Admissions Council.
• Yamamoto, K., Shin, H. J. and Khorramdel, L. (2019), Introduction of multistage adaptive testing design in PISA 2018.(OECD Education Working Papers, No. 209). OECD Publishing: Paris. DOI: 10.1787/b9435d4b-en.
• Yan, D., Lewis, C & von-Davier, A. A. (2014). Multistage test design and scoring with small sample. In D. Yan, A. A. von-Davier & C. Lewis (Eds.), Computerized multistage testing (p. 303–324). CRC Press; Taylor&Francis Group.
• Zenisky, A., Hambleton, R. K. & Luecht, R. M. (2010). Multistage testing: Issues, design, and research. In W. J. van der Linden & C. A.W. Glas (Eds.), Elements of adaptive testing (p. 355-372). Springer.