COMPARISON OF IRT MODELS WITH DIFFERENT GUESSING PARAMETERS

Year 2021, Volume: 12 Issue: 1, 82 - 97, 30.06.2021

Fatih Orçan

Abstract

Abstract: The Three-Parameter Logistic (3PL) model have some advantages over the other Item Response Theory (IRT) models for multiple-choice testing. Under the 3PL model, an examinee with no knowledge can correctly answer an item at the probability of the value of the c-parameter. The propensity for the guessing effect is the same for all ability levels under 3PL models. However, the idea of ability-based guessing has been asserted. In this study, different IRT models for which the guessing parameters are considered in different ways were elaborated. Also, the IRT models were compared with each other via a simulation study and an empirical data set. The results were compared based on item parameter estimation bias and RMSE. Based on the results, the FG3PL model gave the worst results (i.e., larger bias and RMSE) compared to other models. C3PL model was fine when the simulated data were generated by the 2PL model, but not by 3PL data.

Keywords

Item response theory, three-parameter logistic (3PL, guessing parameter

References

• Akour, M. & Al-Omari, H. (2013), “Empirical investigation of the stability of IRT item-parameters estimation”, International Online Journal of Educational Sciences, 5(2): 291-301.
• Burton, R. F. (2002), “Misinformation, partial knowledge and guessing in true ⁄ false tests”, Medical Education, 36: 805-811.
• Cai, L., Thissen, D., & du Toit, S. H. C. (2011), IRTPRO for windows [Computer software]. Lincolnwood, IL: Scientific Software International.
• Chalmers, R., P., (2012), “mirt: A Multidimensional item response theory package for the R environment”, Journal of Statistical Software, 48(6): 1-29.
• Chiu, T. & Camilli, G. (2013), “Comment on 3PL adjustment for guessing”, Applied Psychological Measurement, 37(1): 76-86.
• de la Torre, J. & Hong, Y., (2010), “Parameter estimation with small sample size a higher-order IRT model approach”, Applied Psychological Measurement, 34(4): 267 - 285. DOI: 10.1177/0146621608329501.
• de Ayala, R.J. (2009), The theory and practice of item response theory. New York, NY: The Guilford Press.
• Embretson, S. E., & Reise, S. P. (2000), Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
• Finch, H., & French, B. F. (2019), A comparison of estimation techniques for IRT models with small samples. Applied Measurement in Education, 32(2): 77-96. DOI: 10.1080/08957347.2019.1577243
• Gao S. (2011), “The exploration of the relationship between guessing and latent ability in IRT models”, PhD thesis, Southern Illinois University at Carbondale, Department of Educational Psychology and Special Education.
• Han, K. T. (2007), “WinGen: Windows software that generates IRT parameters and item responses”, Applied Psychological Measurement, 31(5): 457-459.
• Han, K. T. (2012), Fixing the c parameter in the three-parameter logistic model. Practical Assessment Research & Evaluation, 17(1): 1-23.
• Hoogland, J. J. & Boomsma, A. (1998), Robustness studies in covariance structure modeling: An overview and a meta-analysis, Sociological Methods & Research, 26: 329-367.
• Lord, F. M. (1968), An analysis of the verbal scholastic aptitude test using Birnbaum’s three- parameter logistic model. Educational and Psychological Measurement, 28, 989 – 1020. doi:10.1177/001316446802800401.
• McCoubrie, P., (2004), “Improving the fairness of multiple-choice questions: a literature review”, Medical Teacher, 26:8, 709-712, DOI: 10.1080/01421590400013495 Paek, I., (2014), “An investigation of the impact of guessing on coefficient α and reliability”, Applied Psychological Measurement, 1-14. DOI: 10.1177/0146621614559516