Investigating Consequences of Using Item Pre-knowledge in Computerized Multistage Testing
Öz
The goal of this study is to determine the effects of test cheating in a scenario where test-takers use item pre-knowledge in the c-MST, and to urge practitioners to take additional precautions to increase test security. In order to investigate the statistical consequences of item pre-knowledge use in the c-MST, three different cheating scenarios were created, in addition to the baseline condition (e.g., no pre-knowledge usage). The findings were compared under 30-item and 60-item test length conditions with 1-3-3 c-MST panel design. A total of thirty cheaters were generated from a normal distribution, and EAP was used as an ability estimation method. The findings were discussed with the evaluation criteria of mean bias, root mean square error, correlation between true and estimated thetas, conditional absolute bias, and conditional root mean square. It was found that using item pre-knowledge severely affected the estimated thetas, and as the number of compromised items increased, the results got worse. It was concluded that item sharing and/or test cheating seriously damage the test scores, test usage, and score interpretations.
Anahtar Kelimeler
Computerized multistage testing,Test cheating,Item pre-knowledge
Kaynakça
- Armstrong, R. D., Jones, D. H., Li, X., & Wu, L. (1996). A study of a network-flow algorithm and a noncorrecting algorithm for test assembly. Applied Psychological Measurement, 20(1), 89-98.
- Baker, F. (1992). Item response theory. New York, NY: Markel Dekker, INC.
- Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick. In statistical theories of mental test scores (pp. 397–479). Reading, MA: Addison-Wesley.
- Bock, R. D., &Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied psychological measurement, 6(4), 431–444.
- Diao, Q., & van der Linden, W. J. (2011). Automated test assembly using lp_solve version 5.5 in R. Applied Psychological Measurement, 35(5) 398–409.
- Foster, D. (2013). Security issues in technology-based testing. Handbook of test security, 39-83.
- Guo, J., Tay, L., &Drasgow, F. (2009). Conspiracies and test compromise: An evaluation of the resistance of test systems to small-scale cheating. International Journal of Testing, 9(4), 283-309.
- ILOG. (2006). ILOG CPLEX 10.0 [User’s manual]. Paris, France: ILOG SA.
- Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, New Jersey: Lawrence Erlbaum Associates.
- Luecht, R. M., &Nungester, R. J. (1998). Some Practical Examples of Computer‐Adaptive Sequential Testing. Journal of Educational Measurement, 35(3), 229–249.