A Guide for More Accurate and Precise Estimations in Simulative Unidimensional IRT Models

Abstract

There is a great deal of research about item response theory (IRT) conducted by simulations. Item and ability parameters are estimated with varying numbers of replications under different test conditions. However, it is not clear what the appropriate number of replications should be. The aim of the current study is to develop guidelines for the adequate number of replications in conducting Monte Carlo simulation studies involving unidimensional IRT models. For this aim, 192 simulation conditions which included four sample sizes, two test lengths, eight replication numbers, and unidimensional IRT models were generated. Accuracy and precision of item and ability parameter estimations and model fit values were evaluated by considering the number of replications. In this context, for the item and ability parameters; mean error, root mean square error, standard error of estimates, and for model fit; M_2, 〖RMSEA〗_2, and Type I error rates were considered. The number of replications did not seem to influence the model fit, it was decisive in Type I error inflation and error prediction accuracy for all IRT models. It was concluded that to get more accurate results, the number of replications should be at least 625 in terms of accuracy of the Type I error rate estimation for all IRT models. Also, 156 replications and above can be recommended. Item parameter biases were examined, and the largest bias values were obtained from the 3PL model. It can be concluded that the increase in the number of parameters estimated by the model resulted in more biased estimates.

Keywords

• Monte carlo simulation study
• Replication
• Unidimensional item response theory models
• Bias estimation
• Type I error inflation

References

1. Ames, A. J., Leventhal, B. C., & Ezike, N. C. (2020). Monte Carlo simulation in item response theory applications using SAS. Measurement: Interdisciplinary Research and Perspectives, 18(2), 55-74. https://doi.org/10.1080/15366367.2019.1689762
2. Babcock, B. (2011). Estimating a noncompensatory IRT model using Metropolis within Gibbs sampling. Applied Psychological Measurement, 35(4), 317 329. http://dx.doi.org/10.1177/0146621610392366
3. Bahry, L. M. (2012). Polytomous item response theory parameter recovery: an investigation of nonnormal distributions and small sample size [Master’s Thesis]. ProQuest Dissertations and Theses Global.
4. Baker, F. B. (1998). An investigation of the item parameter recovery characteristics of a Gibbs sampling procedure. Applied Psychological Measurement, 22(2), 153-169. https://doi.org/10.1177/01466216980222005
5. Baldwin, P. (2011). A strategy for developing a common metric in item response theory when parameter posterior distributions are known. Journal of Educational Measurement, 48(1), 1-11. Retrieved December 9, 2020, from http://www.jstor.org/stable/23018061
6. Barış Pekmezci, F., & Gülleroğlu, H. (2019). Investigation of the orthogonality assumption in the bifactor item response theory. Eurasian Journal of Educational Research, 19(79), 69-86. http://dx.doi.org/10.14689/ejer.2019.79.4
7. Bulut, O., & Sünbül, Ö. (2017). Monte Carlo simulation studies in item response theory with the R programming language. Journal of Measurement and Evaluation in Education and Psychology, 8(3), 266-287. https://doi.org/10.21031/epod.305821
8. Cai, L., & Thissen, D. (2014). Modern Approaches to Parameter Estimation in Item Response Theory from: Handbook of Item Response Theory Modeling, Applications to Typical Performance Assessment. Routledge.

9. Chalmers, R. P. (2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1 29. https://doi.org/10.18637/JSS.V048.I06
10. Chuah, S. C., Drasgow, F., & Luecht, R. (2006). How big is big enough? Sample size requirements for CAST item parameter estimation. Applied Measurement in Education, 19(3), 241-255. https://doi.org/10.1207/s15324818ame1903_5
11. Cohen, A. S., Kim, S. H., & Baker, F. B. (1993). Detection of differential item functioning in the graded response model. Applied Psychological Measurement, 17(4), 335-350. https://doi.org/10.1177/014662169301700402
12. Crişan, D. R., Tendeiro, J. N., & Meijer, R. R. (2017). Investigating the practical consequences of model misfit in unidimensional IRT models. Applied Psychological Measurement, 41(6), 439-455. https://doi.org/10.1177/0146621617695522
13. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando: Harcourt Brace Jovanovich Inc.
14. De Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford Press.
15. De La Torre, J., & Patz, R. J. (2005). Making the most of what we have: A practical application of multidimensional item response theory in test scoring. Journal of Educational and Behavioral Statistics, 30(3), 295-311. https://www.jstor.org/stable/3701380
16. DeMars, C. E. (2002, April). Recovery of graded response and partial credit parameters in multilog and parscale. Annual meeting of American Educational Research Association, Chicago. https://commons.lib.jmu.edu/cgi/viewcontent.cgi?article=1034&context=gradpsych
17. Feinberg, R. A., & Rubright, J. D. (2016). Conducting simulation studies in psychometrics. Educational Measurement: Issues and Practice, 35(2), 36-49. https://doi.org/10.1111/emip.12111
18. Fu, J. (2019). Maximum marginal likelihood estimation with an expectation–maximization algorithm for multigroup/mixture multidimensional item response theory models (No. RR-19-35). ETS Research Report Series, https://doi.org/10.1002/ets2.12272
19. Gao, F., & Chen, L. (2005). Bayesian or non-Bayesian: A comparison study of item parameter estimation in the three-parameter logistic model. Applied Measurement in Education, 18(4), 351-380. https://doi.org/10.1207/s15324818ame1804_2
20. Glass, G. V., Peckham, P. D., & Sanders, J. R. (1972). Consequences of failure to meet assumptions underlying the fixed effects analyses of variance and covariance. Review of Educational Research, 42(3), 237-288. https://doi.org/10.3102/00346543042003237
21. Goldman, S. H., & Raju, N. S. (1986). Recovery of one-and two-parameter logistic item parameters: An empirical study. Educational and Psychological Measurement, 46(1), 11-21. https://doi.org/10.1177/0013164486461002
22. Hair, J. F., Black W. C., Babin, B. J., & Anderson, R. E. (2019). Multivariate data analysis. (8th edition). Annabel Ainscow.
23. Han, K. T. (2007). WinGen: windows software that generates irt parameters and item responses. Applied Psychological Measurement, 31(5), 457 459. https://doi.org/10.1177/0146621607299271
24. Hanson, B. A. (1998, October). IRT parameter estimation using the EM algorithm. http://www.b-a-h.com/papers/note9801.pdf
25. Harwell, M. (1997). Analyzing the results of monte carlo studies in item response theory. Educational and Psychological Measurement, 57(2), 266-279. https://doi.org/10.1177/0013164497057002006
26. Harwell, M. R., & Baker, F. B. (1991). The use of prior distributions in marginalized Bayesian item parameter estimation: A didactic. Applied Psychological Measurement, 15(4), 375–389. https://doi.org/10.1177/014662169101500409
27. Harwell, M., Stone, C. A., Hsu, T. C., & Kirisci, L. (1996). Monte carlo studies in item response theory. Applied Psychological Measurement, 20(2), 101 125. https://doi.org/10.1177/014662169602000201
28. Hulin, C. L., Lissak, R. I., & Drasgow, F. (1982). Recovery of two and three-parameter logistic item characteristic curves: A Monte Carlo study. Applied Psychological Measurement, 6(3), 249–260. https://doi.org/10.1177/014662168200600301
29. Jiang, S., Wang, C., & Weiss, D. J. (2016). Sample size requirements for estimation of item parameters in the multidimensional graded response model. Frontiers in psychology, 7, 109. https://doi.org/10.3389/fpsyg.2016.00109
30. Kirisci, L., Hsu, T. C., & Yu, L. (2001). Robustness of item parameter estimation programs to assumptions of unidimensionality and normality. Applied Psychological Measurement, 25(2), 146-162. https://doi.org/10.1177/01466210122031975
31. Kleijnen, J. P. (1987). Statistical tools for simulation practitioners. Marcel Dekker.
32. Lee, S., Bulut, O., & Suh, Y. (2017). Multidimensional extension of multiple indicators multiple causes models to detect DIF. Educational and Psychological Measurement, 77(4), 545–569. https://doi.org/10.1177/0013164416651116
33. Lord, F. M. (1968). An analysis of the verbal scholastic aptitude test using Birnbaum's three-parameter logistic model. Educational and Psychological Measurement, 28(4), 989-1020. https://doi.org/10.1177/001316446802800401
34. Matlock, K. L., & Turner, R. (2016). Unidimensional IRT item parameter estimates across equivalent test forms with confounding specifications within dimensions. Educational and Psychological Measurement, 76(2), 258 279. https://doi.org/10.1177/0013164415589756
35. Matlock Cole, K., & Paek, I. (2017). PROC IRT: A SAS procedure for item response theory. Applied Psychological Measurement, 41(4), 311 320. https://doi.org/10.1177/0146621616685062
36. McDonald, R. P. (1982). Linear Versus Models in Item Response Theory. Applied Psychological Measurement, 6(4), 379 396. https://doi.org/10.1177/014662168200600402
37. Mislevy, R. J., & Stocking, M. L. (1989). A consumer's guide to LOGIST and BILOG. Applied Psychological Measurement, 13(1), 57 75. https://doi.org/10.1177/014662168901300106
38. Mooney, C. Z. (1997). Monte Carlo simulation. Thousand Oaks, CA: Sage.
39. Mundform, D. J., Schaffer, J., Kim, M. J., Shaw, D., Thongteeraparp, A., & Supawan, P. (2011). Number of replications required in Monte Carlo simulation studies: A synthesis of four studies. Journal of Modern Applied Statistical Methods, 10(1), 19-28. https://doi.org/10.22237/jmasm/1304222580
40. Park, Y. S., Lee, Y. S., & Xing, K. (2016). Investigating the impact of item parameter drift for item response theory models with mixture distributions. Frontiers in Psychology, 7, 255. https://doi.org/10.3389/fpsyg.2016.00255
41. Patsias, K., Sheng, Y., & Rahimi, S. (2009, September 24-26). A high performance Gibbs sampling algorithm for item response theory. 22nd International Conference on Parallel and Distributed Computing and Communication Systems, Kentucky, USA.
42. Patsula, L. N., & Gessaroli, M. E. A (1995, April). Comparison of item parameter estimates and iccs produced. https://files.eric.ed.gov/fulltext/ED414333.pdf
43. Preecha, C. (2004). Numbers of replications required in ANOVA simulation studies [Doctoral dissertation, University of Northern Colorado]. ProQuest Dissertations and Theses Global.
44. Reise, S. P., & Yu, J. (1990). Parameter recovery in the graded response model using MULTILOG. Journal of Educational Measurement, 27(2), 133 144. https://www.jstor.org/stable/1434973
45. Reise, S., Moore, T., & Maydeu-Olivares, A. (2011). Target rotations and assessing the impact of model violations on the parameters of unidimensional item response theory models. Educational and Psychological Measurement, 71(4), 684-711. https://doi.org/10.1177/0013164410378690
46. Roberts, J. S., Donoghue, J. R., & Laughlin, J. E. (2002). Characteristics of MML/EAP parameter estimates in the generalized graded unfolding model. Applied Psychological Measurement, 26(2), 192-207. https://doi.org/10.1177/01421602026002006
47. Rubinstein, R. Y. (1981). Simulation and the Monte Carlo method. John Wiley and Sons, New York. https://doi.org/10.1002/9780470316511
48. Sahin, A., & Anil, D. (2017). The effects of test length and sample size on item parameters in item response theory. Educational Sciences: Theory & Practice, 17(1), 321-33. https://doi.org/10.12738/estp.2017.1.0270
49. Sarkar, D. (2008). Lattice: multivariate data visualization with R. Springer, New York.
50. Schumacker, R. E, Smith, R. M., & Bush, J. M. (1994, April). Examining replication effects in Rasch fit statistics. American Educational Research Association Annual Meeting, New Orleans.
51. Sen, S., Cohen, A. S., & Kim, S. H. (2016). The impact of non-normality on extraction of spurious latent classes in mixture IRT models. Applied Psychological Measurement, 40(2), 98-113. https://doi.org/10.1177/0146621615605080
52. Sheng, Y., & Wikle, C. K. (2007). Comparing multiunidimensional and unidimensional item response theory models. Educational and Psychological Measurement, 67(6), 899-919. https://doi.org/10.1177/0013164406296977
53. Tavares, H. R., Andrade, D. F. D., & Pereira, C. A. D. B. (2004). Detection of determinant genes and diagnostic via item response theory. Genetics and Molecular Biology, 27(4), 679-685. https://doi.org/10.1590/S1415-47572004000400033
54. Thissen, D., & Wainer, H. (1982). Some standard errors in item response theory. Psychometrika, 47(4), 397-412. https://doi.org/10.1007/BF02293705
55. Thompson, B. (2004). Exploratory and confirmatory factor analysis. Amer Psychological Assn.
56. Thompson, B. (2006). Foundations of behavioral statistics: An insight-based approach. Guilford Press.
57. Thompson, N. A. (2009). Ability estimation with item response theory. Assessment Systems Corporation. https://assess.com/docs/Thompson_(2009)_ _Ability_estimation_with_IRT.pdf
58. Şengül Avşar, A., & Tavşancıl, E. (2017). Examination of polytomous items’ psychometric properties according to nonparametric item response theory models in different test conditions. Educational Sciences: Theory & Practice, 17(2). https://doi.org/10.12738/estp.2017.2.0246
59. van der Linden, W. J. (Ed.). (2018). Handbook of item response theory, three volume set. CRC Press.
60. van Onna, M. J. H. (2004). Ordered latent class models in nonparametric item response theory. [Doctoral dissertation]. University of Groningen.
61. Weissman, A. (2013). Optimizing information using the EM algorithm in item response theory. Annals of Operations Research, 206(1), 627 646. https://doi.org/10.1007/s10479-012-1204-4
62. Yang, S. (2007). A comparison of unidimensional and multidimensional RASCH models using parameter estimates and fit indices when assumption of unidimensionality is violated [Doctoral dissertation, The Ohio State University]. ProQuest Dissertations and Theses Global.
63. Yen, W. M. (1987). A comparison of the efficiency and accuracy of BILOG and LOGIST. Psychometrika, 52(2), 275-291. https://doi.org/10.1007/BF02294241
64. Zhang, B. (2008). Application of unidimensional item response models to tests with items sensitive to secondary dimensions. The Journal of Experimental Education, 77(2), 147-166. https://doi.org/10.3200/JEXE.77.2.147-166