Effects of dimensionality and covariate on items with DIF in mixture models

Abstract

The aim of this study is to determine the differential item functioning (DIF) with a mixture model when the data set is multidimensional. The differences in determining the number of items with DIF and the source of DIF according to the status of considering dimensionality and adding the covariate to the analysis were examined. In this context, a total of 28 items of mathematics and science answered by 7965 individuals in the 3rd booklet of the electronic Trends in International Mathematics and Science Study (eTIMSS) 2019 were found to have a multidimensional structure, and the variable with the highest correlation with the data structure was determined and included in the model as a covariate. In order to select the most appropriate models for the data set, models with different numbers of latent classes belonging to the mixture model and multidimensional mixture model including the covariate were compared. Descriptive statistics of the latent classes created with the selected models were created, item parameters were examined and DIF analysis were conducted. In the light of the findings, it was determined that the number of items with DIF decreased as the model became more complex. In the model with the best knowledge criterion index, it was found that the items with DIF at the knowing level generally differed in favor of the focal group, while the items with DIF at the application and reasoning levels differed in favor of the reference group.

Keywords

• Mixture model
• Multidimensionality
• Covariate variable
• Differential item functioning
• Source of DIF

References

1. Ackerman, T.A. (1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29, 67-91. https://doi.org/10.1111/j.1745-3984.1992.tb00368.x
2. Adams, R., Wilson, M., & Wang, W.-C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1 23. https://doi.org/10.1177/0146621697211001
3. Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716-723. https://doi.org/10.1109/TAC.1974.1100705
4. Altun, M. (2008). Matematik öğretimi [Math teaching]. Aktüel.
5. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (Eds.). (2014). Standards for educational and psychological testing. American Educational Research Association.
6. Baker, F. (2001). The basics of item response theory. ERIC Clearinghouse on Assessment and Evaluation
7. Bulut, O., & Suh, Y. (2017). Detecting multidimensional differential item functioning with the multiple indicators multiple causes model, the item response theory likelihood ratio test, and logistic regression. Frontiers in Education, 2, 1 14. https://doi.org/10.3389/feduc.2017.00051
8. Chen, H.F., & Jin, K.Y. (2018). Applying logistic regression to detect differential item functioning in multidimensional data. Frontiers in Psychology, 9, 1 11. https://doi.org/10.3389/fpsyg.2018.01302

9. Cho, S.J. (2007). A multilevel mixture IRT model for DIF analysis [Unpublished Doctoral Dissertation]. University of Georgia.
10. Cho, S.J., Cohen, A.S., & Kim, S.H. (2013). Markov chain monte carlo estimation of a mixture Rasch model. Journal of Statistical Computation and Simulation, 83, 278 306. https://doi.org/10.1080/00949655.2011.603090
11. Choi, Y.J., Alexeev, N., & Cohen, A.S. (2015) Differential item functioning analysis using a mixture 3-parameter logistic model with a covariate on the TIMSS 2007 mathematics test, International Journal of Testing, 15(3), 239 253, https://doi.org/10.1080/15305058.2015.1007241
12. Choi, I.H., & Wilson, M. (2015). Multidimensional classification of examinees using the mixture random weights linear logistic test model. Educational and Psychological Measurement, 75(1), 78–101. https://doi.org/10.1177/0013164414522124
13. Cohen, A.S., & Bolt, D.M. (2005). A mixture model analysis of differential item functioning. Journal of Educational Measurement, 42, 133 148. https://doi.org/10.1111/j.1745 3984.2005.00007
14. Cohen, A.S., Gregg, N., & Deng, M. (2005). The role of extended time and item content on a high-stakes mathematics test. Learning Disabilities Research and Practice, 20, 225-233.
15. Crawford, A.V., Green, B.S., Levy, R., Lo, W.J., Scott, L., Svetina, D., & Thompson, M.S. (2010). Evaluation of parallel analysis methods for determining the number of factors. Educational and Psychological Measurements, 70(6), 885 901. https://doi.org/10.1177/0013164410379332
16. Çepni, Z., & Kelecioğlu, H. (2021). Detecting differential item functioning using SIBTEST, MH, LR and IRT methods. Journal of Measurement and Evaluation in Education and Psychology, 12(3), 267-285. https://doi.org/10.21031/epod.988879
17. Dai, Y. (2009). A mixture Rasch model with a covariate: A simulation study via Bayesian Markov Chain Monte Carlo estimation [Doctoral dissertation, University of Maryland]. Digital Repository at the University of Maryland. http://hdl.handle.net/1903/9926
18. Dai, Y. (2013). A mixture Rasch model with a covariate: A simulation study via Bayesian Markov chain Monte Carlo estimation. Applied Psychological Measurement, 37, 375-396. https://doi.org/10.1177/0146621612475076
19. Dayton, C.M. (1999). Latent class scaling analysis. Sage university paper series on quantitative applications in the social sciences. Sage.
20. De Ayala, R.J., & Santiago, S.Y. (2017). An introduction to mixture item response theory models. Journal of School Psychology, 60, 25-40. https://doi.org/10.1016/j.jsp.2016.01.002
21. DeMars, C.E., & Lau, A. (2011). Differential item functioning detection with latent classes: how accurately can we detect who is responding differentially? Educational and Psychological Measurement, 71(4), 597-616. https://doi.org/10.1177/0013164411404221
22. Diaz, E., Brooks, G., & Johanson, G. (2021). Detecting differential item functioning: Item response theory methods versus the Mantel-Haenszel procedure. International Journal of Assessment Tools in Education 8(2), 376–393. https://doi.org/10.21449/ijate.730141
23. Dinno, A. (2018). paran: Horn's test of principal components/factors. R package version 1.5.2, https://CRAN.R-project.org/package=paran
24. Doğan, Ö., & Atar, B. (2024). Comparing differential item functioning based on multilevel mixture item response theory, mixture item response theory and manifest groups. Journal of Measurement and Evaluation in Education and Psychology, 15(2), 120 137. https://doi.org/10.21031/epod.1457880
25. Embretson, S.E. (2007). Mixed Rasch models for measurement in cognitive psychology. In M. Von Davier, & C.H. Carstensen (Eds.), Multivariate and mixture distribution Rasch models: extensions and applications (pp. 235-253). Springer
26. Finch, W.H., & Hernández Finch, M.E. (2013). Investigation of specific learning disability and testing accommodations based differential item functioning using a multilevel multidimensional mixture item response theory model. Educational and Psychological Measurement, 73(6), 973-993. https://doi.org/10.1177/0013164413494776
27. Fraenkel, J.R., Wallen, N.E., & Hyun, H.H. (2012). How to design and evaluate research in education (8th ed.). Mc Graw Hill.
28. Frick, H., Strobl, C., & Zeileis, A. (2015). Rasch Mixture Models for DIF Detection: A Comparison of Old and New Score Specifications. Educational and psychological measurement, 75(2), 208–234. https://doi.org/10.1177/0013164414536183
29. Fukuhara, H., & Kamata, A. (2011). A bifactor multidimensional item response theory model for differential item functioning analysis on testlet-based items. Applied Psychological Measurement 35(8) 604–622. https://doi.org/10.1177/0146621611428447
30. Gelin, M.N. (2005). Type I error rates of the DIF MIMIC approach using Joreskog’s covariance matrix with ML and estimation [Unpublished doctoral dissertation]. The University of British Columbia.
31. Gierl, M.J., Jodoin, M.G., & Ackerman, T.A. (2000). Performance of Mantel-Haenszel, Simultaneous Item Bias Test, and Logistic Regression when the proportion of DIF items is large [Paper presentation]. The Annual Meeting of the American Educational Research Association, New Orleans, LA.
32. Gomez-Benito, J., & Navas-Ara, M.J. (2000). A comparison of χ2, RFA and IRT based procedures in the detection of DIF. Quality and Quantity, 34(1), 17 31. https://doi.org/10.1023/A:1004703709442
33. Gürdil, H. (2023). The use of multidimensional item response theory models in controlling differential item functioning [Unpublished doctoral dissertation]. Ankara University.
34. Hagenaars, J.A., & McCutcheon, A.L. (Eds.). (2002). Applied latent class analysis. Cambridge University Press.
35. Hartig, J., & Höhler, J. (2009). Multidimensional IRT models for the assessment of competencies. Studies in Educational Evaluation, 35(2 3), 57 63. https://doi.org/10.1016/j.stueduc.2009.10.002
36. Horn, J.L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrica, 30(2), 179-185. https://doi.org/10.1007/BF02289447
37. Huang, X. (2010). Differential item functioning: The consequence of language, curriculum, or culture? [Unpublished doctoral dissertation]. UC Berkeley.
38. Kaiser, H.F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141 151. https://doi.org/10.1177/001316446002000116
39. Kane, M.T. (2006). Validation. In R.B. Brennan (Ed.), Educational Measurement (4th ed., pp. 17-64). American Council on Education/Praeger.
40. Karadavut, T., Cohen, A.S., & Kim, S.H. (2019). Mixture rasch model with main and interaction effects of covariates on latent class membership. International Journal of Assessment Tools in Education, 6(3), 362-377. https://doi.org/10.21449/ijate.592789
41. Koo, T.K., & Li, M.Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
42. Kristanjansonn, E., Aylesworth, R., McDowell, I., & Zumbo, B.D. (2005). A Comparison of four methods for detecting differential item functioning in ordered response model. Educational and Psychological Measurement, 65(6), 935 953. https://doi.org/10.1177/0013164405275668
43. Li. T. (2014). Different approaches to covariate inclusion in the mixture Rasch model [Unpublished Doctoral dissertation]. University of Maryland.
44. Li, F., Cohen, A.S., Kim, S.H., & Cho, S.J. (2009). Model selection methods for mixture dichotomous IRT models. Applied Psychological Measurement, 33(5), 353 373. https://doi.org/10.1177/0146621608326422
45. Li, T., Jiao, H., & Macready, G.B. (2016). Different approaches to covariate inclusion in the mixture Rasch model. Educational and Psychological Measurement, 76(5), 848-872. https://doi.org/10.1177/0013164415610380
46. Liaw, Y.L. (2015). When can multidimensional item response theory (MIRT) models be a solution for differential item functioning (DIF)? A Monte Carlo Simulation Study [Unpublished doctoral dissertation]. University of Washington.
47. Maij-de Meij, A.M., Kelderman, H., & van der Flier, H. (2010). Improvement in detection of differential item functioning using a mixture item response theory model. Multivariate Behavioral Research, 45(6), 975-999. https://doi.org/10.1080/00273171.2010.533047
48. McCutcheon A.C. (1987). Latent class analysis. Sage.
49. McLachlan, G., & Peel, D. (2000). Finite mixture models. John Wiley.
50. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
51. Mislevy, R.J. (1993). Foundations of a new test theory. In N. Frederiksen, R.J. Mislevy, and I.I. Bejar (Eds.), Test theory for a new generation of tests. Erlbaum.
52. Muthén, B. (1997). Latent variable modeling of longitudinal and multilevel data. Sociological Methodology, 27, 453–480. http://www.jstor.org/stable/271113
53. Muthén, L.K., & Muthén, B.O. (2017). Mplus: Statistical Analysis with Latent Variables: User’s Guide (Version 8). Authors.
54. Oberski, D.L. (2016). Mixture models: Latent profile and latent class analysis. In J. Robertson, & M. Kaptein (Eds.), Modern statistical methods for HCI (pp. 275-287). ( Human–Computer Interaction Series). Springer. https://doi.org/10.1007/978-3-319-26633-6_12
55. Pektaş, S. (2018). The effects of differential item functioning determination methods on test parameters estimates, decision studies, g and phi coefficients [Unpublished doctoral dissertation]. Gazi University.
56. Penfield, R.D., & Camilli, G. (2007). Differential item functioning and item bias. In C. R. Rao & S. Sinharay (Eds.), Handbook of Statistics (pp. 125-167). Elsevier Science & Technology. https://doi.org/10.1016/S0169-7161(06)26005-X
57. Piccone, A.V. (2009). A comparison of three computational procedures for solving the number of factors problem in exploratory factor analysis [Unpublished Doctoral Dissertation]. University of Northern Colorado. https://digscholarship.unco.edu/dissertations/228/
58. Pine, S.M. (1977). Applications of item characteristic curve theory to the problem of test bias. In D.J. Weiss (Ed.), Applications of computerized adaptive testing: Proceedings of a symposium presented at the 18th Annual Convention of the Military Testing Association (Vol. 77, No. 1, pp. 37–43). University of Minnesota, Department of Psychology, Psychometric Methods Program.
59. Polya, G. (1957) How to solve it. a new aspect of mathematical method (2nd ed.). Princeton University Press.
60. R Core Team. (2024). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/
61. Reckase, M.D. (2009). Multidimensional item response theory. Springer.
62. Reise, S.P., & Revicki, D.A. (Eds.). (2015). Handbook of item response theory modeling: Applications to typical performance assessment. Routledge/Taylor & Francis Group.
63. Samuelsen, K. (2005). Examining differential item functioning from a latent class perspective [Doctoral dissertation, University of Maryland, College Park]. ProQuest Dissertations and Theses Global.
64. Schwarz, G. (1978). Estimating the Dimension of a Model. The Annals of Statistics, 6(2), 461–464. http://www.jstor.org/stable/2958889
65. Sclove, L. (1987) Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52, 333-343. http://dx.doi.org/10.1007/BF02294360
66. Sen, S., & Cohen, A.S. (2019). Applications of mixture IRT models: A literature review. Measurement: Interdisciplinary Research and Perspectives, 17(4), 177–191. https://doi.org/10.1080/15366367.2019.1583506
67. Sırgancı, G. (2019). The effect of covariant variable on determination of differential item functioning using mixture rasch model [Unpublished doctoral dissertation]. Ankara University.
68. Smit, A., Kelderman, H., & van der Flier, H. (1999). Collateral information and mixed rasch models. Methods of Psychological Research Online, 4(3), 1-13.
69. Smit, A., Kelderman, H., & van der Flier, H. (2000). The mixed Birnbaum model: Estimation using collateral information. Methods of Psychological Research Online, 5, 31-43.
70. Tabachnick, B.G., & Fidell, L.S. (2013). Using multivariate statistics (6th ed.). Pearson.
71. Tay, L., Newman, D.A., & Vermunt, J.K. (2011). Using mixed-measurement item response theory with covariates (MM-IRT-C) to ascertain observed and unobserved measurement equivalence. Organizational Research Methods, 14(1), 147 176. https://doi.org/10.1177/1094428110366037
72. Uyar, Ş. (2015). Comparing differential item functioning based on manifest groups and latent classes [Unpublished doctoral dissertation]. Hacettepe University.
73. Uysal Saraç, M. (2022). Comparison of covariate including approaches to mixture rasch model in terms of classification and estimation accuracy [Unpublished doctoral dissertation]. Gazi University.
74. Vrieze, S.I. (2012). Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17, 228-243. https://doi.org/10.1037/a0027127
75. Walker, C.M., & Beretvas, S.N. (2001). An empirical investigation demonstrating the multidimensional DIF paradigm: a cognitive explanation for DIF. Journal of Educational Measurement, 38(2), 147–163. https://doi.org/10.1111/j.1745-3984.2001.tb01120.x
76. Yalçın, S. (2018). Determining differential item functioning with the mixture item response theory. Eurasian Journal of Educational Research, 18(74), 187-206.
77. Yanagida, T. (2023). misty: Miscellaneous functions for statistical analysis and data management (Version 0.5.2) [Computer software]. https://doi.org/10.32614/CRAN.package.misty
78. Yao, L., & Li, F. (2015). A DIF detection procedure in multidimensional item response theory framework using MCMC technique. International Journal of Quantitative Research in Education, 2(3), 285-304.
79. Yıldıztekin, B. (2020). The investigation of test dimension effect to differential item functioning under different conditions [Unpublished doctoral dissertation]. Hacettepe University.
80. Zhang, Y. (2017). Detection of latent differential item functioning (DIF) using mixture 2PL IRT model with covariate [Unpublished doctoral dissertation]. University of Pittsburgh.
81. Zhu, X. (2013). Distinguishing continuous and discrete approaches to multilevel mixture IRT models: A model comparison perspective [Unpublished doctoral dissertation]. University of Maryland.
82. Zieky, M. (1993). Practical questions in the use of DIF statistics in test development. In P.W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 337–347). Lawrence Erlbaum Associates, Inc.