Computer Adaptive Testing Simulations in R

Abstract

Computer adaptive testing is an important research field in educational measurement, and simulation studies play a critically important role in CAT development and evaluation. Both Monte Carlo and Post Hoc simulations are frequently used in CAT studies in order to investigate the effects of different factors on test efficiency and to compare different test designs. Although there are several softwares for CAT simulations, R is preferred since it includes many free packages and gives researchers opportunity to write their own functions according to their own requirements besides being free. The purpose of this study is to make an introduction and demonstration of how to use catR package in CAT simulations. Different examples were provided in the context of this study and R codes were presented with explanations. Then, the output files were briefly explained. It is thought that this paper is helpful for the researchers who are interested in CAT simulations.

Keywords

• computer adaptive testing
• simulation
• R programming language

Computer Adaptive Testing Simulations in R

Öz

Computer adaptive testing is an important research field in educational measurement, and simulation studies play a critically important role in CAT development and evaluation. Both Monte Carlo and Post Hoc simulations are frequently used in CAT studies in order to investigate the effects of different factors on test efficiency and to compare different test designs. Although there are several softwares for CAT simulations, R is preferred since it includes many free packages and gives researchers opportunity to write their own functions according to their own requirements besides being free. The purpose of this study is to make an introduction and demonstration of how to use catR package in CAT simulations. Different examples were provided in the context of this study and R codes were presented with explanations. Then, the output files were briefly explained. It is thought that this paper is helpful for the researchers who are interested in CAT simulations.

Anahtar Kelimeler

• Computer adaptive testing
• Simulation
• R programming language

References

1. Bulut, O. & Sünbül, Ö. (2017). R programlama dili ile madde tepki kuramında monte carlo simülasyon çalışmaları [Monte carlo simulation studies in item response theory with the R programming language]. Journal of Measurement and Evaluation in Education and Psychology, 8(3), 266-287. DOI: 10.21031/epod.305821
2. Choi, S. W. (2009). Firestar: Computerized adaptive testing simulation program for polytomous item response theory models. Applied Psychological Measurement, 33(8), 644-645.
3. Embretson, S. E. & Reise, S. P. (2000). Item response theory for psychologists. Mahwah N.J.: L. Erlbaum Associates.
4. Han, K. T. (2012). SimulCAT: Windows software for simulating computerized adaptive test administration. Applied Psychological Measurement, 36(1), 64-66.
5. Han, K. T. & Kosinski, M. (2014). Software tools for multistage testing simulations. In D. Yan, A. A. von-Davier & C. Lewis (Eds.), Computerized multistage testing: Theory and applications (p. 411–420). CRC Press: Taylor&Francis Group.
6. Hendrickson, A. (2007). An NCME instructional module on multistage testing. Educational Measurement: Issues and Practice, Summer 2007, 44-52.
7. Lee, S., Choi, Y.J., & Cohen, A. (2018). Automating simulation research for item response theory using R. International Journal of Assessment Tools in Education, 5(4), 682-700. Retrieved from http://ijate.net/index.php/ijate/article/view/596
8. Lord, F. M. (1984). Standard errors of measurement at different ability levels. ETS Research Reports, 1984(1), I 11. Retrieved from https://onlinelibrary.wiley.com/doi/epdf/10.1002/j.2330-8516.1984.tb00048.x

9. Magis D. & Raiche G. (2011). catR: An R package for computerized adaptive testing. Applied Psychological Measurement, 35, 576-577.
10. Magis, D., Yan, D. & von-Davier, A. (Eds.). (2017). Computerized adaptive and multistage testing with R: Using packages catr and mstr. Switzerland: Springer.
11. R Core Team. (2014). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved from http://www.R-project.org/
12. Thissen, D. & Mislevy, R. J. (2000). Testing algorithms. In H. Wainer (Ed.), Computerized adaptive testing: A primer (2. ed.). Mahwah N.J.: Lawrence Erlbaum Associates.
13. Thompson, N. A. (2007). A practitioner’s guide for variable-length computerized classification testing. Practical Assessment Research & Evaluation, 12(1). Retrieved from http://pareonline.net/getvn.asp?v=12&n=1
14. Wainer, H. (2000). Introduction and history. In H. Wainer (Ed.), Computerized adaptive testing: A primer (2.ed., p. 1–22). Mahwah N.J.: Lawrence Erlbaum Associates.
15. Wainer, H. & Mislevy, R. J. (2000). Item response theory, item calibration, and proficiency estimation. In H. Wainer (Ed.), Computerized adaptive testing: A primer (2. ed.). Mahwah N.J.: Lawrence Erlbaum Associates.
16. Wang, K. (2017). A fair comparison of the performance of computerized adaptive testing and multistage adaptive testing (Doctoral Dissertation). Michigan State University.
17. Weiss, D. J. & Kingsbury, G. G. (1984). Application of computer adaptive testing to educational problems. Journal of Educational Measurement, 21(4), 361–375. https://doi.org/10.1111/j.1745-3984.1984.tb01040.x
18. Weiss, D. J. & Guyer, R. (2010). Manual for CATSim: Comprehensive simulation of computerized adaptive testing. St. Paul MN: Assessment Systems Corporation.
19. Yan, D., von-Davier, A. A. & Lewis, C. (2014b). Overview of computerized multistage tests. In D. Yan, A. A. von-Davier & C. Lewis (Eds.), Computerized multistage testing. CRC Press: Taylor&Francis Group.