The Efficacy of the IRTree Framework for Detecting Missing Data Mechanisms in Educational Assessments

Abstract

The effectiveness of methods for handling missing data in educational assessments depends on understanding the underlying missing mechanisms. This study investigates the performance of the IRTree framework in detecting missing data mechanisms using a Monte Carlo simulation. Omitted responses were simulated at varying proportions according to three mechanisms: MCAR, MAR, and MNAR, across tests with different lengths and sample sizes. The IRTree was employed to model the omitted responses and detect the mechanisms based on the correlations between the propensity to omit and proficiency. Results indicate that the IRTree accurately identifies all three missing data mechanisms, with no relationship between propensity to omit and proficiency under MCAR, and negative correlations for MAR, reaching up to -0.3, and for MNAR, as high as -0.8. Furthermore, the detection of MAR and MNAR mechanisms became more pronounced with higher proportions of omitted responses, longer tests, and larger sample sizes. IRTree framework not only enables educators and researchers to accurately understand the nature of missing data but also guides them in using appropriate methods for handling it.

Keywords

• IRTree
• missing data
• missing mechanism
• simulation
• R language

References

1. Alagöz, Ö. E. C., & Meiser, T. (2023). Investigating heterogeneity in response strategies: A mixture multidimensional IRTree approach. Educational and Psychological Measurement, 84(5), 957-993. https://doi.org/10.1177/00131644231206765
2. Alarcon, G. M., Lee, M. A., & Johnson, D. (2023). A Monte Carlo study of IRTree models' ability to recover item parameters. Frontiers In Psychology, 14, 1003756. https://doi.org/10.3389/fpsyg.2023.1003756
3. Allison, P. D. (2002). Missing data. Sage Publications.
4. Baker, F. B. (2001). The basics of item response theory. ERIC Clearinghouse on Assessment and Evaluation.
5. Bock, R. D., & Aitkin M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4), 443-459. https://doi.org/10.1007/BF02293801
6. Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431-444. https://doi.org/10.1177/014662168200600405
7. Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-460). MA: Addison-Wesley.
8. Böckenholt, U. (2012). Modeling multiple response processes in judgment and choice. Psychological Methods, 17(4), 665-678. https://doi.org/10.1037/a0028111

9. Böckenholt, U. (2017). Measuring response styles in Likert items. Psychological methods, 22(1), 69–83. https://doi.org/10.1037/met0000106
10. Chalmers, R. P. (2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
11. Cheema, J. R. (2014). A review of missing data handling methods in education research. Review of Educational Research, 84(4), 487-508. https://doi.org/10.3102/0034654314532697
12. Collins, L. M., Schafer, J. L., & Kam, C. M. (2001) A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6(4), 330-51.
13. Damiani, V. (2016). Large-scale assessments and educational policies in Italy. Research Papers in Education, 31(5), 529–541.
14. De Ayala, R. J., Plake, B. S., & Impara, J. C. (2001). The impact of omitted responses on the accuracy of ability estimation in item response theory. Journal of Educational Measurement, 38(3), 213–234. https://doi.org/10.1111/j.1745-3984.2001.tb01124.x
15. De Boeck, P., & Partchev, I. (2012). IRTrees: Tree based item response models of the GLMM family. Journal of Statistical Software, 48, 1-28. https://doi.org/10.18637/jss.v048.c01
16. Debeer, D., Janssen, R., & De Boeck, P. (2017). Modeling skipped and not-reached items using IRTrees. Journal of Educational Measurement, 54(3), 333-363. https://doi.org/10.1111/jedm.12147
17. DeMars, C. (2010). Item response theory: Understanding statistics measurement. Oxford University Press.
18. Dibek, M. I. (2019). Examination of the extreme response style of students using IRTree: The case of TIMMS 2015. International Journal of Assessment Tools in Education, 6, 300-313. https://doi.org/10.21449/ijate.534118
19. Enders, C. K. (2010). Applied missing data analysis. The Guilford Press.
20. Feinberg, R. A., & Rubright, J. D. (2016). Conducting simulation studies in psychometrics. Educational Measurement: Issues and Practice, 35(2), 36-49. https://doi.org/10.1111/emip.12111
21. Glas, C. A. W., & Pimentel, J. L. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 48(6), 907-922. https://doi.org/10.1177/0013164408315262
22. Glas, C. A. W., Pimentel, J. L., & Lamers, S. M. A. (2015). Nonignorable data in IRT models: Polytomous models with covariates. Psychological Test and Assessment Modeling, 57(4), 523-541.
23. Graham, J. W. (2012). Missing data analysis and design. Springer. Hambleton, R. K., Swaminathan, H. & Rogers, H. J. (1991). Fundamentals of item response theory. California: Sage Publications.
24. Harwell, M., Stone, C. A., Hsu, T. C., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20(2), 101-125. https://doi.org/10.1177/014662169602000201
25. Holman, R., & Glas, C. A. (2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Pyschology, 58, 1-17. https://doi.org/10.1111/j.2044-8317.2005.tb00312.x
26. Huang, H. Y. (2020). A mixture IRTree model for performance decline and nonignorable missing data. Educational and Psychological Measurement, 80(6), 1168-1195. https://doi.org/10.1177/0013164420914711
27. Huisman, M. (2000). Imputation of missing item responses: Some simple techniques. Quality & Quantity, 34, 331–351. https://doi.org/10.1023/A:1004782230065
28. Jeon, M., & De Boeck, P. (2016). A generalized item response tree model for psychological assessments. Behavior Research Methods, 48, 1070-1085. https://doi.org/10.3758/s13428-015-0631-y
29. Jeon, M., De Boeck, P., & van der Linden, W. (2017). Modeling answer change behavior: An application of a generalized item response tree model. Journal of Educational and Behavioral Statistics, 42(4), 467-490. https://doi.org/10.3102/1076998616688015
30. Jeon, M., Rijmen, F. & Rabe-Hesketh, S. (2014). Flexible item response theory modeling with FLIRT. Applied Psychological Measurement, 38, 404-405. https://doi.org/10.1177/0146621614524982
31. Jin, K.-Y., Wu, Y.-J., & Chen, H.-F. (2022). A new multiprocess IRT model with ideal points for likert-type items. Journal of Educational and Behavioral Statistics, 47(3), 297-321. https://doi.org/10.3102/10769986211057160
32. Köhler, C., Pohl, S., & Carstensen, C. (2017). Dealing with item nonresponse in large‐scale cognitive assessments: The impact of missing data methods on estimated explanatory relationships. Journal of Educational Measurement, 54, 397-419. https://doi.org/10.1111/jedm.12154
33. Leventhal, B. C. (2019). Extreme response style: A simulation study comparison of three multidimensional item response models. Applied Psychological Measurement, 43(4), 322–335. https://doi.org/10.1177/0146621618789392
34. Little, R. J. A., & Rubin, D. B. (1987). Statistical analysis with missing data. John Wiley & Sons.
35. Little, R. J. A. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83, 1198–1202.
36. Little, T. D., Lang, K. M., Wu, W., & Rhemtulla, M. (2016). Developmental psychopathology. In D. Cicchetti (Ed.), Missing Data (pp. 760-797). John Wiley & Sons.
37. Martens, K., Niemann, D., & Teltemann, J. (2016). Effects of international assessments in education – a multidisciplinary review. European Educational Research Journal, 15(5), 516-522. https://doi.org/10.1177/1474904116668886
38. McKnight, P. E., McKnight, K. M., Sidani, S. & Figueredo, A. J. (2007). Missing data: A gentle introduction. Guilford Press.
39. Newman, D. A. (2014). Missing data: Five practical guidelines. Organizational research methods, 17(4), 372-411. https://doi.org/10.1177/1094428114548590
40. Park, M., & Wu, A. D. (2019). Item response tree models to investigate acquiescence and extreme response styles in Likert-type rating scales. Educational and Psychological Measurement, 79(5), 911–930. https://doi.org/10.1177/0013164419829855
41. Peugh, J. L., & Enders, C. K. (2004). Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of Educational Research, 74(4), 525-556. https://doi.org/10.3102/00346543074004525
42. Pigott, T. D. (2010). A review of methods for missing data. Educational Research and Evaluation: An International Journal on Theory and Practice, 7(4), 353-383. https://doi.org/10.1076/edre.7.4.353.8937
43. Plieninger, H. (2021). Developing and applying Ir-Tree models: Guidelines, caveats, and an extension to multiple groups. Organizational Research Methods, 24(3), 654-670. https://doi.org/10.1177/1094428120911096
44. Pohl, S., Gräfe, L. & Rose, N. (2014). Dealing with omitted and not-reached items in competence tests: Evaluating approaches accounting for missing responses in item response theory models. Educational and Psychological Measurement, 74(3), 423–452. https://doi.org/10.1177/0013164413504926
45. Quirk, V. L., & Kern, J. L. (2023). Using IRTree models to promote selection validity in the presence of extreme response styles. Journal of Intelligence, 11(11), 216. https://doi.org/10.3390/jintelligence11110216
46. Rose, N., von Davier, M., & Nagengast, B. (2015). Modeling omitted and not-reached items in IRT models. Psychometrika, 82, 795-819. https://doi.org/10.1007/s11336-016-9544-7
47. Rose, N., von Davier, M., & Xu, X. (2010). Modeling nonignorable missing data with item response theory (IRT) (ETS Research Report No. RR-10-11). Educational Testing Service.
48. Roth, P. L. (1994). Missing data: A conceptual review for applied psychologists. Personnel Psychology, 47(3), 537–560. https://doi.org/10.1111/j.1744-6570.1994.tb01736.x.
49. Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581-592. https://doi.org/10.1093/biomet/63.3.581
50. Spratto, E. M., Leventhal, B. C., & Bandalos, D. L. (2021). Seeing the forest and the trees: Comparison of two IRTree models to investigate the impact of full versus endpoint-only response option labeling. Educational and Psychological Measurement, 81(1), 39-60. https://doi.org/10.1177/0013164420918655
51. Sulis, I., & Porcu, M. (2017). Handling missing data in item response theory. Assessing the accuracy of a multiple imputation procedure based on latent class analysis. Journal of Classification, 34, 327–359. https://doi.org/10.1007/s00357-017-9220-3
52. Tabachnick, B. G., & Fidell L. S. (2007). Using multivariate statistics. Allyn and Bacon.