Item Response Theory Assumptions: A Comprehensive Review of Studies with Document Analysis

Abstract

Item Response Theory (IRT), over its nearly 100-year history, has become one of the most popular methodologies for modeling response patterns in measures in education, psychology and health. Due to its advantages, IRT is particularly popular in large-scale assessments. A pre-condition for the validity of the estimations obtained from IRT is that the data meet the model assumptions. The purpose of this study is to examine the testing of model assumptions in studies using IRT models. For this purpose, 107 studies in the National Thesis Center of the Council of Higher Education that use the IRT model on real data were examined. The studies were analyzed according to sample size, unidimensionality, local independence, overall model fit, item fit and non-speedness test criteria. According to the results, it was observed that the unidimensionality assumption was tested at a high level (89%) and Factor Analytic approaches were predominantly used. Local independence assumption was not tested in 36% of the studies, unidimensionality was cited as evidence in 40% of the studies and tested in 24% of the studies. Overall model fit was tested at a moderate level (51%) and Log-Likelihood and information criteria were used. Item fit and Non-Speedness testing were tested at a low level (26% and 9%). IRT assumptions should be considered as a whole and all assumptions should be tested from an evidence-based perspective.

Keywords

• Item response theory
• assumption
• unidimensionality
• local independence
• overall model fit
• Item fit
• non-speedness test
• factor analysis.

References

1. Antoniou, F., Alkhadim, G., Mouzaki, A., & Simos, P. (2022). A Psychometric Analysis of Raven’s Colored Progressive Matrices: Evaluating Guessing and Carelessness Using the 4PL Item Response Theory Model. Journal of Intelligence 10(1),6 MDPI AG. https://doi.org/10.3390/jintelligence10010006
2. Ary, D., Jacobs, L. C., Sorensen, C., & Razavieh, A. (2010). Introduction to research in education (Eight). Belmont: wadsworth Cengage Learning.
3. Aybek, E. C. (2023). The relation of item difficulty between Classical Test Theory and Item Response Theory: Computerized Adaptive Test perspective. Egitimde ve Psikolojide Olcme ve Degerlendirme Dergisi, 14(2), 118–127. https://doi.org/10.21031/epod.1209284
4. Baker, F. B. (2001). The basics of item response theory. For full text: http://ericae. net/irt/baker..
5. Bock, R.D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.
6. Boduroğlu, E., & Anil, D. (2023). Examining group differences in mathematics achievement: Explanatory item response model application. OPUS Toplum Araştırmaları Dergisi, 20(53), 385–395. https://doi.org/10.26466/opusjsr.1226914
7. Brzezińska, J. (2016). A polytomous item response theory models using R / Politomiczne modele teorii odpowiedzi na pozycje testowe w programie R. Ekonometria. https://doi.org/10.15611/ekt.2016.2.04
8. Chalmers, R., P. (2012). mirt: A multidimensional ıtem response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29. doi: https://doi.org/10.18637/jss.v048.i06

9. Chen, W.-H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics: A Quarterly Publication Sponsored by the American Educational Research Association and the American Statistical Association, 22(3), 265–289. https://doi.org/10.3102/10769986022003265
10. Chou, Y.-T., & Wang, W.-C. (2010). Checking dimensionality in ıtem response models with principal component analysis on standardized residuals. In educational and psychological measurement (Vol. 70, Issue 5, pp. 717–731). SAGE Publications. https://doi.org/10.1177/0013164410379322
11. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Holt, Rinehart and Winston, 6277 Sea Harbor Drive, Orlando, FL 32887.
12. De Ayala, R. J. (2013). The theory and practice of item response theory. Guilford Publications.
13. De Boeck, P., & Wilson, M. (Eds.). (2004). Explanatory item response models: a generalized linear and nonlinear approach. New York: Springer.
14. DeMars, C. (2010). Item response theory. Oxford University Press.
15. Dogan, N., & Basokcu, T. O. (2010). İstatistik tutum ölçegi için uygulanan faktör analizi ve asamalı kümeleme analizi sonuçlarının karsilastirilmasi. Journal of Measurement and Evaluation in Education and Psychology, 1(2), 65-71.
16. Doğan, Ö., & Atar, B. (2024). Comparing differential item functioning based on multilevel mixture item response theory, mixture item response theory and manifest groups. Egitimde ve Psikolojide Olcme ve Değerlendirme Dergisi, 15(2), 120–137. https://doi.org/10.21031/epod.1457880
17. Drasgow, F., Levine, M. V., & Williams, E. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. The British Journal of Mathematical and Statistical Psychology, 38(1), 67–86. https://doi.org/10.1111/j.2044-8317.1985.tb00817.x
18. Eckes, T. (2011). Introduction to many-facet Rasch measurement. Frankfurt am Main: Peter Lang.
19. Edwards, M. C., Houts, C. R., & Cai, L. (2018). A diagnostic procedure to detect departures from local independence in item response theory models. Psychological Methods, 23(1), 138–149. https://doi.org/10.1037/met0000121
20. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Psychology Press.
21. Erkuş, A. (2006). Sınıf öğretmenleri için ölçme ve değerlendirme: kavramlar ve uygulamalar. Ekinoks Yayınları, Ankara.
22. Erkuş, A., Sünbül, Ö., Sünbül, S. Ö., Yormaz, S., & Aşiret, S. (2017). psikolojide ölçme ve ölçek geliştirme-II ölçme araçlarının psikometrik nitelikleri ve ölçme kuramları. Pegem Yayınları, Ankara.
23. Fan, X. (1998). Item response theory and classical test theory: An empirical comparison of their ıtem/person statistics. Educational and Psychological Measurement, 58(3). 357–381. https://doi.org/10.1177/0013164498058003001
24. Gökçen Ayva Yörü, F. (2024). Thematic and metadological analysis of doctoral dissertations on measurement and. International Journal of Education Technology and Scientific Researches. https://doi.org/10.35826/ijetsar.721
25. Gulliksen, H. (1950). The reliability of speeded tests. Psychometrika, 15(3), 259-269.
26. Hambleton, R. K. (1989). Principles and selected applications of item response theory. In R. L. Linn (Ed.), Educational measurement. Washington, DC: American Council on Education and Macmillan.
27. Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), 38-47.
28. Hambleton, R. K., & Swaminathan, H. (1985). A look at psychometrics in the Netherlands.
29. Hambleton, R. K., & Swaminathan, H. (1985). Assumptions of item response theory. Item response theory, 15-31. Springer, Dordrecht.
30. Hambleton, R. K., Shavelson, R. J., Webb, N. M., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory (Vol. 2). Sage.
31. Harwell, M. R., & Janosky, J. E. (1991). An Empirical Study of the Effects of Small Datasets and Varying Prior Variances on Item Parameter Estimation in BILOG. Applied Psychological Measurement, 15(3), 279–291. https://doi.org/10.1177/014662169101500308
32. Hasan, B. M. S., & Abdulazeez, A. M. (2021). A review of principal component analysis algorithm for dimensionality reduction. Journal of Soft Computing and Data Mining, 2(1), 20-30. https://doi.org/10.30880/jscdm.2021.02.01.003
33. Himelfarb, I. (2019). A primer on standardized testing: History, measurement, classical test theory, item response theory, and equating. The Journal of Chiropractic Education, 33(2), 151–163. https://doi.org/10.7899/jce-18-22
34. Hulin, C. L., Lissak, R. I., & Drasgow, F. (1982). Recovery of two- and three-parameter logistic item characteristic curves: A Monte Carlo study. Applied Psychological Measurement, 6(3), 249-260.
35. Junker, B. W. (1991). Essential independence and likelihood-based ability estimation for polytomous items. Psychometrika, 56(2), 255-278.
36. Kartal, S., & Mor Dı̇rlı̇k, E. (2021). Examining the dimensionality and monotonicity of an attitude dataset based on the item response theory models. International Journal of Assessment Tools in Education, 8(2), 296–309. https://doi.org/10.21449/ijate.728362
37. Kılıç, A. F., Koyuncu, İ., & Uysal, İ. (2023). Scale development based on item response theory: A systematic review. International Journal of Psychology and Educational Studies, 10(1), 209–223. https://doi.org/10.52380/ijpes.2023.10.1.982
38. Koyuncu, İ., & Kılıç, A. F. (2019). The use of exploratory and confirmatory factor analyses: A document analysis. TED EĞİTİM VE BİLİM. https://doi.org/10.15390/eb.2019.7665
39. Linacre JM. (2009). Local independence and residual covariance: a study of olympic figure skating ratings. Journal of Applied Measurement, 10(2), 157-69.
40. Looney, M. A., & Spray, J. A. (1992). Effects of violating local independence on IRT parameter estimation for the binomial trials model. Research Quarterly For Exercise and Sport, 63(4), 356-359.
41. Lord, F. M. (1953). The relation of test score to the trait underlying the test. Educational And Psychological Measurement, 13(4), 517-549.
42. Lord, F. M. (1968). An Analysis of the Verbal Scholastic Aptitude Test using Birnbaum's three-parameter logistic model. Educational and Psychological Measurement, 28, 989-1020.
43. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores.
44. Lumsden, J. (1961). The construction of unidimensional tests. Psychological Bulletin, 58(2), 122–131. https://doi.org/10.1037/h0048679
45. Maydeu-Olivares, A., & Joe, H. (2006). Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 71(4), 713–732. https://doi.org/10.1007/s11336-005-1295-9
46. McDonald, R. P. (1981). The dimensionality of tests and items. British Journal of mathematical and statistical Psychology, 34(1), 100-117.
47. Min, S., & He, L. (2014). Applying unidimensional and multidimensional item response theory models in testlet-based reading assessment. Language Testing, 31(4), 453–477. https://doi.org/10.1177/0265532214527277
48. Mutluer, C., & Çakan, M. (2023). Comparison of test equating methods based on Classical Test Theory and Item Response Theory. Uludağ Üniversitesi Eğitim Fakültesi Dergisi, 36(3), 866–906. https://doi.org/10.19171/uefad.1325587
49. Nandakumar, R., & Stout, W. (1993). Refinements of Stout’s procedure for assessing latent trait unidimensionality. Journal of Educational Statistics, 18(1), 41–68. https://doi.org/10.3102/10769986018001041
50. Novick, M. R. (1966). The axioms and principal results of classical test theory. Journal Of Mathematical Psychology, 3(1), 1-18.
51. Orlando, M., & Thissen, D. (2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24(1), 50–64. https://doi.org/10.1177/01466216000241003
52. Reckase, M. D. (1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 21(1), 25-36.
53. Reckase, M. D. (2009). Multidimensional item response theory. Springer, New York, NY.
54. Ree, M. J., & Jensen, H. E. (1980). Effects of sample size on linear equating of item characteristic curve parameters. D. J. Weiss (Ed.), Proceedings of the 1979 computerized adaptive testing conference. Minneapolis: University of Minnesota.
55. Reise, S. P., & Waller, N. G. (2009). Item response theory and clinical measurement. Annual review of clinical psychology, 5(1), 27-48.
56. Rijmen, F., Tuerlinckx, F., De Boeck, P., & Kuppens, P. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8(2), 185.
57. Rupp, A. A., & Zumbo, B. D. (2006). Understanding parameter invariance in unidimensional IRT models. Educational and Psychological Measurement, 66(1), 63–84. https://doi.org/10.1177/0013164404273942
58. Saatcioglu, F. M., & Sen, S. (2023). The analysis of TIMSS 2015 data with confirmatory mixture item response theory: A multidimensional approach. International Journal of Testing, 23(4), 257–275. https://doi.org/10.1080/15305058.2023.2214648
59. Selçuk, E., & Demir, E. (2024). Comparison of item response theory ability and item parameters according to classical and Bayesian estimation methods. International Journal of Assessment Tools in Education, 11(2), 213–248. https://doi.org/10.21449/ijate.1290831
60. Shepard, L., Camilli, G., & Williams, D. M. (1984). Accounting for statistical artifacts in item bias research. Journal of Educational Statistics, 9(2), 93–128. https://doi.org/10.3102/10769986009002093
61. Sireci, S. G. (1991, June). Sample independent item parameters?An investigation of the stability of IRT item parameters estimated from small data sets. Paper presented at the annual Conference of Northeastern Educational Research Association, New York, NY.
62. Sözer, E., & Kahraman, N. (2021). Investigation of psychometric properties of likert items with same categories using polytomous item response theory models. Egitimde ve Psikolojide Olcme ve Degerlendirme Dergisi, 12(2), 129–146. https://doi.org/10.21031/epod.819927
63. Stone, C. A. (2000). Monte Carlo based null distribution for an alternative goodness‐of‐fit test statistic in IRT models. Journal of Educational Measurement, 37(1), 58–75. https://doi.org/10.1111/j.1745-3984.2000.tb01076.x
64. Stone, C. A., & Zhu, X. (2015). Bayesian analysis of item response theory models using SAS. SAS Institute Inc.
65. Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52(4), 589-617.
66. Suhr, D. (2006). Exploratory or Confirmatory Factor Analysis? Statistics and Data Analysis, 1–17.
67. Swaminathan, H., & Gifford, J. A. (1983). Esimation of parameters in the three-parameter latent trait model. In D. J. Weiss (Ed.), New horizons in testing, (pp. 9-30). New York: Academic Press.
68. Swineford F. (1956). Technical Manual for Users of Test Analyses, Statistical Report, 56-42. Princeton, NJ: Educational Testing Service.
69. Şahı̇n, M. G., Yildirim, Y., & Boztunc Öztürk, N. (2023). Examining the achievement test development process in the educational studies. Participatory Educational Research, 10(1), 251–274. https://doi.org/10.17275/per.23.14.10.1
70. Thissen, D., & Steinberg, L. (2020). An intellectual history of parametric item response theory models in the twentieth century. Chinese/English Journal of Educational Measurement and Evaluation, 1(1). https://doi.org/10.59863/gpml7603
71. Trabin, T. E., & Weiss, D. J. (1983). The person response curve: Fit of individuals to item response theory models. New horizons in testing (pp. 83-108). Academic press.
72. Tucker, L. R., Humphreys, L. G., & Roznowski, M. A. (1986). Comparative accuracy of five ındices of dimensionality of binary ıtems. van der Linden, W. J. (2010). Elements of adaptive testing. C. A. Glas (Ed.). New York, NY: Springer.
73. Wainer, H., Bradlow, E. T., & Wang, X. (2007). Testlet response theory and its applications. Cambridge University Press.
74. Watkins, M. W. (2006). Determining parallel analysis criteria. Journal of Modern Applied Statistical Methods, 5(2), 344-346.
75. Wright, B., & Panchapakesan, N. (1969). A procedure for sample-free item analysis. Educational And Psychological Measurement, 29(1), 23-48.
76. Yen, W. M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30(3), 187–213. https://doi.org/10.1111/j.1745-3984.1993.tb00423.x
77. Yiğiter, M. S., & Doğan, N. (2023). Comparison of different computerized adaptive testing approaches with shadow test under different test length and ability estimation method conditions. Egitimde ve Psikolojide Olcme ve Degerlendirme Dergisi, 14(4), 396–412. https://doi.org/10.21031/epod.1202599
78. Zanon, C., Hutz, C. S., Yoo, H., & Hambleton, R. K. (2016). An application of item response theory to psychological test development. Psicologia: Reflexão e Crítica, 29(1). https://doi.org/10.1186/s41155-016-0040-x
79. Ziegler, M., & Hagemann, D. (2015). Testing the Unidimensionality of Items. European Journal of Psychological Assessment, 31(4), 231–237. https://doi.org/10.1027/1015-5759/a000309
80. Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99(3), 432.