The Effect of Strong and Weak Unidimensional Item Pools on Computerized Adaptive Classification Testing
Abstract
Computerized Adaptive Classification Tests (CACT) aim to classify individuals effectively with high classification accuracy and few items over large item pools. The characteristic features of the item pool include the number of items, item factor loadings, the distribution of the Test Information Function, and dimensionality. In this study, we present the results of a comprehensive simulation study that was examined how item selection methods (MFI-KLI), ability estimation methods (EAP-WLE) and classification methods (SPRT-CI) were affected by strong and weak unidimensional item pools. Findings of the study indicate that CI had always produced results with classification accuracy similar to SPRT but with a test length of almost half. Additionally, KLI and MFI item selection methods were not affected by the item pool characteristic as weak or strong unidimensionality. From findings of this study, it can be recommended to use CI with EAP in CACT studies, whether the item pool is weak or strong unidimensional, but WLE only under strong unidimensional item pools. Additionally, EAP and SPRT methods are recommended to prefer in the weak unidimensional item pool.
Keywords
Computerized adaptive classification testing, unidimesionality, strong unidimensionality, weak unidimensionality, item pool
References
- Ayan, C. (2018). Comparing the psychometric features of traditional and computerized adaptive classification test in the cognitive diagnostic model [Unpublished Doctoral Dissertation]. Ankara University.
- Aybek, E. C. (2016). An investigation of applicability of the self assessment inventory as a computerized adaptive test (CAT) [Unpublished Doctoral Dissertation]. Ankara University.
- Breslow, N. E., & Holubkov, R. (1997). Weighted likelihood, pseudo-likelihood and maximum likelihood methods for logistic regression analysis of two-stage data. Statistics in Medicine, 16, 103-116.
- Cheng, P. E., & Liou, M. (2000). Estimation of trait level in computerized adaptive testing. Applied Psychological Measurement, 24(3), 257–265.
- Demir, S. (2019). Investigation of classification accuracy at computerized adaptive classification tests [Unpublished Doctoral Dissertation]. Hacettepe University.
- Diao, Q., & Reckase, M. (2009). Comparison of ability estimation and item selection methods in multidimensional computerized adaptive testing. In D. J. Weiss (Ed.), Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing (pp. 1-13). UMI Research Press.
- Dogan, N., Soysal, S., & Karaman, H. (2017). Aynı örnekleme açımlayıcı ve doğrulayıcı faktör analizi uygulanabilir mi?[ Can exploratory and confirmatory factor analysis be conducted to the same sample?]. In Ö. Demirel & S. Dinçer (Eds.), Küreselleşen dünyada eğitim [Education in a globalizing World] (pp. 373- 400). Pegema Publishing.
- Doğruöz, E. (2018). Investigation of adaptive multistage test based on test assembly methods [Unpublished Doctoral Dissertation]. Hacetteepe University.
- Dooley, K. (2002), Simulation research methods. In J. Baum (Ed.), Companion to organizations, (pp. 829-848). Blackwell.
- Eggen, T. J. H. M. (1999). Item selection in adaptive testing with the sequential probability ratio test. Applied Psychological Measurement, 23, 249-260.