Model misspecification in multidimensional IRT: Consequences of ignoring the guessing parameter

Samet Eker; Melike Kara Şahin; Hülya Kelecioğlu

Model misspecification in multidimensional IRT: Consequences of ignoring the guessing parameter

Abstract

This study examined the effect of ignoring the guessing parameter in the context of multidimensional Item Response Theory on item and ability parameter estimation, using Monte Carlo simulation. Data generated from a two-dimensional 3PL model (N = 2000; 40 items) were estimated under each condition using both the correctly specified model (2D-3PL) and the misspecified model (2D-2PL) via Bayesian methods (Stan/MCMC). The inter-dimensional correlation was manipulated at ρ = .10, .50, and .80. Parameter recovery was evaluated using bias, root-mean-square error (RMSE), and estimated-true correlations. Results indicated that the 2D-2PL model systematically underestimated the discrimination parameter across all conditions (16–21% negative bias), and that this distortion intensified as the correlation increased. The difficulty parameter also showed downward bias (bias ≈ −0.24 to −0.27), leading items to appear easier than they actually were. In terms of error magnitude, the 2PL/3PL RMSE ratio for discrimination ranged from 2.32 to 2.47 and increased rather than decreased under high-correlation conditions. Ability estimates, by contrast, were found to be relatively robust to model misspecification, with the 2PL–3PL RMSE difference remaining at approximately 0.5%–1.0%. The results show that 3PL should be preferred for multidimensional multiple-choice tests with N ≥ 2000, and that using 2PL for computational convenience may lead to substantial measurement errors, particularly in applications such as item calibration, item selection, and test equating.

Keywords

References

Ackerman, T.A., Gierl, M.J., & Walker, C.M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22(3), 37-51. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
Alahmadi, S., & DeMars, C.E. (2025). From item estimates to test operations: The cascading effect of rapid guessing. Journal of Educational Measurement, 62(4), 740 762. https://doi.org/10.1111/jedm.70010
Baker, F.B., & Kim, S.-H. (Eds.). (2004). Item response theory: Parameter estimation techniques, second edition (2nd ed.). CRC Press. https://doi.org/10.1201/9781482276725
Bandalos, D.L., & Leite, W. (2013). Use of Monte Carlo studies in structural equation modeling research. In G. R. Hancock & R.O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 625-666). Information Age Publishing.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F.M. Lord & M.R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-479). Addison-Wesley.
Cai, L. (2017). flexMIRT version 3.5: Flexible multilevel multidimensional item analysis and test scoring [Computer software]. Vector Psychometric Group.
Carpenter, B., Gelman, A., Hoffman, M.D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017). Stan: A Probabilistic Programming Language. Journal of Statistical Software, 76(1). https://doi.org/10.18637/jss.v076.i01
Chalmers, R.P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06

Cuhadar, I., Yang, Y., & Paek, I. (2021). Consequences of ignoring guessing effects on measurement invariance analysis. Applied Psychological Measurement, 45(4), 283 296. https://doi.org/10.1177/01466216211013915
DeMars, C.E. (2016). Partially compensatory multidimensional item response theory models: Two alternate model forms. Educational and Psychological Measurement, 76(2), 231-257. https://doi.org/10.1177/0013164415589595
DeMars, C.E., & Jurich, D.P. (2015). The interaction of ability differences and guessing when modeling differential item functioning with the Rasch model: Conventional and tailored calibration. Educational and Psychological Measurement, 75(4), 610 633. https://doi.org/10.1177/0013164414554082
Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates.
Feuerstahler, L.M. (2018). Sources of error in IRT trait estimation. Applied Psychological Measurement, 42(5), 359–375. https://doi.org/10.1177/0146621617733955
Fu, Y., Strachan, T., Ip, E.H., Willse, J.T., Chen, S.-H., & Ackerman, T. (2020). The recovery of correlation between latent abilities using compensatory and noncompensatory multidimensional IRT models. International Journal of Testing, 20(2), 169 186. https://doi.org/10.1080/15305058.2019.1692212
Gao, F., & Chen, L. (2005). Bayesian or non-Bayesian: A comparison study of item parameter estimation in the three-parameter logistic model. Applied Measurement in Education, 18(4), 351–380. https://doi.org/10.1207/s15324818ame1804_2
Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., & Rubin, D.B. (2013). Bayesian Data Analysis (3rd ed.). Chapman and Hall/CRC. https://doi.org/10.1201/b16018
Hambleton, R.K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Kluwer Academic Publishers. https://doi.org/10.1007/978-94-017-1988-9
Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory. Sage Publications.
Harwell, M.R., & Janosky, J.E. (1991). An empirical study of the effects of small datasets and varying prior variances on item parameter estimation in BILOG. Applied Psychological Measurement, 15(3), 279–291. https://doi.org/10.1177/014662169101500308
Harwell, M., Stone, C.A., Hsu, T.-C., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20(2), 101-125. https://doi.org/10.1177/014662169602000201
Hosseinzadeh, M., & Cole, K.L.M. (2024). Effects of the quantity and magnitude of cross-loading and model specification on MIRT item parameter recovery. Educational and Psychological Measurement, 84(5), 1012–1040. https://doi.org/10.1177/00131644231210509
Hulin, C.L., Lissak, R.I., & Drasgow, F. (1982). Recovery of two- and three-parameter logistic item characteristic curves: A monte carlo study. Applied Psychological Measurement, 6(3), 249–260. https://doi.org/10.1177/014662168200600301
Ip, E.H. (2010). Empirically indistinguishable multidimensional IRT and locally dependent unidimensional item response models. British Journal of Mathematical and Statistical Psychology, 63(2), 395–416. https://doi.org/10.1348/000711009X466835
Kolen, M.J., & Brennan, R.L. (2014). Test equating, scaling, and linking: Methods and practices (3rd ed.). Springer Science + Business Media. https://doi.org/10.1007/978-1-4939-0317-7
Lewandowski, D., Kurowicka, D., & Joe, H. (2009). Generating random correlation matrices based on vines and the extended onion method. Journal of Multivariate Analysis, 100(9), 1989-2001. https://doi.org/10.1016/j.jmva.2009.04.008
Lord, F. (1952). A theory of test scores (Psychometric Monograph No. 7). Psychometric Corporation. https://www.psychometricsociety.org/sites/main/files/file-attachments/mn07.pdf?1576607452
Lord, F.M. (1980). Applications of item response theory to practical testing problems (1st ed.). Routledge. https://doi.org/10.4324/9780203056615
Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Addison-Wesley.
Morris, T.P., White, I.R., & Crowther, M.J. (2019). Using simulation studies to evaluate statistical methods. Statistics in Medicine, 38(11), 2074–2102. https://doi.org/10.1002/sim.8086
Natesan, P., Nandakumar, R., Minka, T., & Rubright, J.D. (2016). Bayesian prior choice in IRT estimation using MCMC and variational Bayes. Frontiers in Psychology, 7, 1422. https://doi.org/10.3389/fpsyg.2016.01422
Nguyen, H.V., & Waller, N.G. (2023). Local minima in multidimensional item response theory models. Multivariate Behavioral Research, 58(1), 147. https://doi.org/10.1080/00273171.2022.2160953
Ohlsson, S. (2011). Deep learning: How the mind overrides experience. Cambridge University Press. https://doi.org/10.1017/CBO9780511780295
Rasch, G. (1960). Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche.
Reckase, M.D. (1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 21(1), 25–36. https://doi.org/10.1177/0146621697211002
Reckase, M.D. (2009). Multidimensional item response theory. Springer. https://doi.org/10.1007/978-0-387-89976-3
Snow, R.E., & Lohman, D.F. (1989). Implications of cognitive psychology for educational measurement. In R.L. Linn (Ed.), Educational measurement (3rd ed., pp. 263-331). Macmillan.
Spray, J.A., Davey, T.C., Reckase, M.D., Ackerman, T.A., & Carlson, J.E. (1990). Comparison of two logistic multidimensional item response theory models (Research Report No. ONR 90-8). ACT, Inc.
Stenhaug, B.A., & Domingue, B.W. (2022). Predictive fit metrics for item response models. Applied Psychological Measurement, 46(2), 136–155. https://doi.org/10.1177/01466216211066603
Strachan, T., Ip, E., Fu, Y., Ackerman, T., Chen, S.H., & Willse, J. (2020). Robustness of projective IRT to misspecification of the underlying multidimensional model. Applied psychological measurement, 44(5), 362–375. https://doi.org/10.1177/0146621620909894
Svetina, D., Valdivia, A., Underhill, S., Dai, S., & Wang, X. (2017). Parameter recovery in multidimensional item response theory models under complexity and nonnormality. Applied Psychological Measurement, 41(7), 530–544. https://doi.org/10.1177/0146621617707507
van der Linden, W.J., & Hambleton, R.K. (1997). Item response theory: Brief history, common models, and extensions. Handbook of Modern Item Response Theory, 1-28. https://doi.org/10.1007/978-1-4757-2691-6_1
von Davier, M., Gonzalez, E., & Mislevy, R.J. (2009). What are plausible values and why are they useful? IERI Monograph Series, 2, 9-36.
Yen, W.M. (1981). Using simulation results to choose a latent trait model. Applied Psychological Measurement, 5(2), 245–262. https://doi.org/10.1177/014662168100500212
Zhao, Y., & Hambleton, R.K. (2017). Practical consequences of item response theory model misfit in the context of test equating with mixed-format test data. Frontiers in Psychology, 8, Article 484. https://doi.org/10.3389/fpsyg.2017.00484
Zoltak, T., & Golonka, K. (2015). Does guessing matter? Differences between ability estimates from 2PL and 3PL IRT models in case of guessing. Edukacja, 4(135), 96–111.

Details

Primary Language

English

Subjects

Measurement Theories and Applications in Education and Psychology

Journal Section

Research Article

Authors

Samet Eker ^*
0000-0001-8266-2953
Türkiye

Melike Kara Şahin
0009-0008-3703-3101
Türkiye

Hülya Kelecioğlu
0000-0002-0741-9934
Türkiye

Early Pub Date

October 28, 2026

Publication Date

-

Submission Date

April 8, 2026

Acceptance Date

October 11, 2026

Published in Issue

Year 2026 Number: Advanced Online Publication

IZ

https://izlik.org/JA76DA49HJ

Cite

RIS / Bibtex

APA

Eker, S., Kara Şahin, M., & Kelecioğlu, H. (2026). Model misspecification in multidimensional IRT: Consequences of ignoring the guessing parameter. International Journal of Assessment Tools in Education, Advanced Online Publication, 1172-1192. https://izlik.org/JA76DA49HJ

AMA

1.Eker S, Kara Şahin M, Kelecioğlu H. Model misspecification in multidimensional IRT: Consequences of ignoring the guessing parameter. Int. J. Assess. Tools Educ. 2026;(Advanced Online Publication):1172-1192. https://izlik.org/JA76DA49HJ

Chicago

Eker, Samet, Melike Kara Şahin, and Hülya Kelecioğlu. 2026. “Model Misspecification in Multidimensional IRT: Consequences of Ignoring the Guessing Parameter”. International Journal of Assessment Tools in Education, no. Advanced Online Publication: 1172-92. https://izlik.org/JA76DA49HJ.

EndNote

Eker S, Kara Şahin M, Kelecioğlu H (October 1, 2026) Model misspecification in multidimensional IRT: Consequences of ignoring the guessing parameter. International Journal of Assessment Tools in Education Advanced Online Publication 1172–1192.

IEEE

[1]S. Eker, M. Kara Şahin, and H. Kelecioğlu, “Model misspecification in multidimensional IRT: Consequences of ignoring the guessing parameter”, Int. J. Assess. Tools Educ., no. Advanced Online Publication, pp. 1172–1192, Oct. 2026, [Online]. Available: https://izlik.org/JA76DA49HJ

ISNAD

Eker, Samet - Kara Şahin, Melike - Kelecioğlu, Hülya. “Model Misspecification in Multidimensional IRT: Consequences of Ignoring the Guessing Parameter”. International Journal of Assessment Tools in Education. Advanced Online Publication (October 1, 2026): 1172-1192. https://izlik.org/JA76DA49HJ.

JAMA

1.Eker S, Kara Şahin M, Kelecioğlu H. Model misspecification in multidimensional IRT: Consequences of ignoring the guessing parameter. Int. J. Assess. Tools Educ. 2026;:1172–1192.

MLA

Eker, Samet, et al. “Model Misspecification in Multidimensional IRT: Consequences of Ignoring the Guessing Parameter”. International Journal of Assessment Tools in Education, no. Advanced Online Publication, Oct. 2026, pp. 1172-9, https://izlik.org/JA76DA49HJ.

Vancouver

1.Samet Eker, Melike Kara Şahin, Hülya Kelecioğlu. Model misspecification in multidimensional IRT: Consequences of ignoring the guessing parameter. Int. J. Assess. Tools Educ. [Internet]. 2026 Oct. 1;(Advanced Online Publication):1172-9. Available from: https://izlik.org/JA76DA49HJ