DIF Analysis with a Multidimensional Multilevel Mixture Model with Covariates

Ömer Doğan; Burcu Atar

doi:10.63612/ijesp.1961623

DIF Analysis with a Multidimensional Multilevel Mixture Model with Covariates

Abstract

This study aims to compare observed groups and latent classes, derived from mixture models, in terms of the number of items displaying differential item functioning (DIF) and their corresponding effect size levels. It also seeks to investigate the nature of DIF, utilizing it as a tool for collecting validity evidence in the testing process. In this direction, a dataset comprising 20 items (10 mathematics and 10 science) from the eTIMSS 2019 booklet 1, which included 22 countries, was created. The dataset structure was analyzed and found to exhibit both multidimensional and multilevel characteristics. Results showed that, for both observed group variables, only items with negligible DIF (Level A) were identified, whereas mixture models revealed many items with DIF at effect Levels B and C. Examination of the selected CB2-C2 model provided a more detailed descriptive picture of item-parameter differences across latent classes, particularly with

Keywords

References

Ackerman, T. A. (1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29, 67–91. https://doi.org/10.1111/j.1745-3984.1992.tb00368.x
Ackerman, T. A. (1994). Using multidimensional item response theory to understand what items and tests are measuring. Applied Measurement in Education, 7, 255–278. https://doi.org/10.1207/s15324818ame0704_1
Ackerman, T. A., Gierl, M. J., & Walker, C. M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22(3), 37–53. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (Eds.). (2014). Standards for educational and psychological testing. American Educational Research Association.
Atar, B. (2010). Basit doğrusal regresyon analizi ile hiyerarşik doğrusal modeller analizinin karşılaştırılması [A comparison of linear regression analysis and hierarchical linear modeling]. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 1(2), 78–84.
Bilir, M. K. (2009). Mixture item response theory-MIMIC model: Simultaneous estimation of differential item functioning for manifest groups and latent classes (Unpublished doctoral dissertation). Florida State University.
Bolt, D. M., Cohen, A. S., & Wollack, J. A. (2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39, 331–348. https://doi.org/10.1111/j.1745-3984.2002.tb01146.x
Boughton, K., & Yamamoto, K. (2007). A hybrid model for test speededness. In M. von Davier & C. H. Carstensen (Eds.), Multivariate and mixture distribution Rasch models (pp. 147–156). Springer.

Ceylan, N. (2022). Uzman görüşüne göre yanlı olduğu belirlenen maddelerin değişen madde fonksiyonu belirleme yöntemleri ile tutarlılığının incelenmesi [Examining the consistency of the items determined to be biased according to expert opinions with differential item functioning] (Unpublished master’s thesis). Gazi University.
Cho, S. J. (2007). A multilevel mixture IRT model for DIF analysis (Unpublished doctoral dissertation). University of Georgia.
Cho, S.-J., & Cohen, A. S. (2010). A multilevel mixture IRT model with an application to DIF. Journal of Educational and Behavioral Statistics, 35(3), 336–370. https://doi.org/10.3102/1076998609353111
Cho, S.-J., Suh, Y., & Lee, W. Y. (2016). An NCME instructional module on latent DIF analysis using mixture item response models. Educational Measurement: Issues and Practice, 35(1), 48–61. https://doi.org/10.1111/emip.12093
Choi, Y. J., Alexeev, N., & Cohen, A. S. (2015). Differential item functioning analysis using a mixture 3-parameter logistic model with a covariate on the TIMSS 2007 mathematics test. International Journal of Testing, 15(3), 239–253. https://doi.org/10.1080/15305058.2015.1007241
Clauser, B. E., & Mazor, K. M. (1998). Using statistical procedures to identify differentially functioning test items: An NCME instructional module. Educational Measurement: Issues and Practice, 17(1), 31–44. https://doi.org/10.1111/j.1745-3992.1998.tb00619.x
Cohen, A. S., & Bolt, D. M. (2005). A mixture model analysis of differential item functioning. Journal of Educational Measurement, 42(2), 133–148. https://doi.org/10.1111/j.1745-3984.2005.00007.x
Cohen, A. S., Gregg, N., & Deng, M. (2005). The role of extended time and item content on a high-stakes mathematics test. Learning Disabilities Research and Practice, 20, 225–233. https://doi.org/10.1111/j.1540-5826.2005.00138.x
Dai, Y. (2009). A mixture Rasch model with a covariate: A simulation study via Bayesian Markov chain Monte Carlo estimation (Unpublished doctoral dissertation). University of Maryland, College Park.
Dai, Y. (2013). A mixture Rasch model with a covariate: A simulation study via Bayesian Markov chain Monte Carlo estimation. Applied Psychological Measurement, 37, 375–396. https://doi.org/10.1177/0146621612475076
De Ayala, R. J., Kim, S.-H., Stapleton, L. M., & Dayton, C. M. (2002). Differential item functioning: A mixture distribution conceptualization. International Journal of Testing, 2, 243–276. https://doi.org/10.1080/15305058.2002.9669495
De Boeck, P., Cho, S.-J., & Wilson, M. (2011). Explanatory secondary dimension modeling of latent differential item functioning. Applied Psychological Measurement, 35(8), 583–603. https://doi.org/10.1177/0146621611428446
De Jong, M. G., & Steenkamp, J. B. E. M. (2010). Finite mixture multilevel multidimensional ordinal IRT models for large scale cross-cultural research. Psychometrika, 75, 3–32. https://doi.org/10.1007/s11336-009-9134-z
DeMars, C. E., & Lau, A. (2011). Differential item functioning detection with latent classes: How accurately can we detect who is responding differentially? Educational and Psychological Measurement, 71(4), 597–616. https://doi.org/10.1177/0013164411404221
Finch, W. H., & Hernández Finch, M. E. (2013). Investigation of specific learning disability and testing accommodations based differential item functioning using a multilevel multidimensional mixture item response theory model. Educational and Psychological Measurement, 73(6), 973–993. https://doi.org/10.1177/0013164413494776
Fukuhara, H., & Kamata, A. (2011). A bifactor multidimensional item response theory model for differential item functioning analysis on testlet-based items. Applied Psychological Measurement, 35(8), 604–622. https://doi.org/10.1177/0146621611428447
Gürkan, G. (2021). From OLS to multilevel multidimensional mixture IRT: A model refinement approach to investigating patterns of relationships in PISA 2012 data (Unpublished doctoral dissertation). Boston College.
Holland, P. W., & Wainer, H. (1993). Differential item functioning. Lawrence Erlbaum.
Hox, J. J. (2013). Multilevel regression and multilevel structural equation modeling. In T. D. Little (Ed.), The Oxford handbook of quantitative methods (pp. 281–294). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199934898.013.0014
Hu, P. G., & Dorans, N. J. (1989). The effects of deleting differentially functioning items on equating functions and reported score distributions. Educational Testing Service.
Jensuttiwetchakul, P., Kanjanawasee, S., & Ngudgratoke, S. (2016). A development of the 3PL MMM-IRT model for identifying latent class. Procedia - Social and Behavioral Sciences, 217, 719–728. https://doi.org/10.1016/j.sbspro.2016.02.132
Jeon, M. (2019). A specialized confirmatory mixture IRT modeling approach for multidimensional tests. Psychological Test and Assessment Modeling, 61(1), 91–123.
Lee, W. Y., Cho, S. J., & Sterba, S. K. (2018). Ignoring a multilevel structure in mixture item response models: Impact on parameter recovery and model selection. Applied Psychological Measurement, 42(2), 136–154. https://doi.org/10.1177/0146621617711999
Li, T. (2014). Different approaches to covariate inclusion in the mixture Rasch model (Unpublished doctoral dissertation). University of Maryland, College Park.
Magis, D., Béland, S., Tuerlinckx, F., & De Boeck, P. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42, 847–862. https://doi.org/10.3758/BRM.42.3.847
Maij-de Meij, A. M., Kelderman, H., & van der Flier, H. (2010). Improvement in detection of differential item functioning using a mixture item response theory model. Multivariate Behavioral Research, 45(6), 975–999. https://doi.org/10.1080/00273171.2010.533047
McLachlan, G., & Peel, D. (2000). Finite mixture models. John Wiley.
Meyer, J. P. (2010). A mixture Rasch model with item response time components. Applied Psychological Measurement, 34, 521–538. https://doi.org/10.1177/0146621609355451
Mislevy, R. J., & Verhelst, N. D. (1990). Modeling item responses when different subjects employ different solution strategies. Psychometrika, 55, 195–215. https://doi.org/10.1007/BF02295283
Potenza, M. T., & Dorans, N. J. (1995). DIF assessment for polytomously scored items: A framework for classification and evaluation. Applied Psychological Measurement, 19(1), 23–37. https://doi.org/10.1177/014662169501900104
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage.
Reckase, M. D. (1985). The difficulty of items that measure more than one ability. Applied Psychological Measurement, 9, 401–412. https://doi.org/10.1177/014662168500900409
Revelle, W. (2023). psych: Procedures for psychological, psychometric, and personality research (R package version 2.3.9). Northwestern University. https://CRAN.R-project.org/package=psych
Rijmen, F., & De Boeck, P. (2005). A relation between a between-item multidimensional IRT model and the mixture Rasch model. Psychometrika, 70, 481–496. https://doi.org/10.1007/s11336-002-1007-7
Rost, J. (1990). Rasch models in latent classes: An integration of two approaches to item analysis. Applied Psychological Measurement, 14(3), 271–282. https://doi.org/10.1177/014662169001400305
Roussos, L. A., & Stout, W. F. (1996). A multidimensionality-based DIF analysis paradigm. Applied Psychological Measurement, 20(4), 355–371. https://doi.org/10.1177/014662169602000404
Samuelsen, K. (2005). Examining differential item functioning from a latent class perspective (Unpublished doctoral dissertation). University of Maryland, College Park.
Samuelsen, K. (2008). Examining differential item function from a latent class perspective. In G. Hancock & K. Samuelsen (Eds.), Mixture models in latent variable research. Information Age Publishing.
Scheuneman, J. (1979). A method of assessing bias in test items. Journal of Educational Measurement, 16, 143–152. https://doi.org/10.1111/j.1745-3984.1979.tb00095.x
Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464. https://doi.org/10.1214/aos/1176344136
Şen, S. (2022). Mplus ile yapısal eşitlik modellemesi uygulamaları. Nobel.
Sırgancı, G. (2019). Karma Rasch model ile değişen madde fonksiyonunun belirlenmesinde kovaryant (ortak) değişkenin etkisi [The effect of covariant variable on determination of differential item functioning using mixture Rasch model] (Unpublished doctoral dissertation). Ankara University.
Smit, A., Kelderman, H., & van der Flier, H. (1999). Collateral information and mixed Rasch models. Methods of Psychological Research Online, 4, 1–13. https://doi.org/10.23668/psycharchives.12745
Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Pearson.
Yıldız, H. (2021). irtGUI: Item response theory analysis with a graphic user interface (R package version 0.2). https://CRAN.R-project.org/package=irtGUI
Zieky, M. (1993). Practical questions in the use of DIF statistics in test development. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 337–347). Lawrence Erlbaum.
Zumbo, B. D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223–233. https://doi.org/10.1080/15434300701375832

Details

Primary Language

English

Subjects

Measurement Theories and Applications in Education and Psychology, Classroom Measurement Practices, Curriculum Evaluation in Education

Journal Section

Research Article

Authors

Ömer Doğan ^*
0000-0001-5169-520X
Türkiye

Burcu Atar
0000-0003-3527-686X
Türkiye

Early Pub Date

August 4, 2026

Publication Date

-

Submission Date

June 1, 2026

Acceptance Date

July 28, 2026

Published in Issue

Year 2026 Number: Advanced Online Publication

DOI

https://doi.org/10.63612/ijesp.1961623

IZ

https://izlik.org/JA34MD35BE

Cite

RIS / Bibtex

APA

Doğan, Ö., & Atar, B. (2026). DIF Analysis with a Multidimensional Multilevel Mixture Model with Covariates. International Journal of Educational Studies and Policy, Advanced Online Publication. https://doi.org/10.63612/ijesp.1961623

AMA

1.Doğan Ö, Atar B. DIF Analysis with a Multidimensional Multilevel Mixture Model with Covariates. IJESP. 2026;(Advanced Online Publication). doi:10.63612/ijesp.1961623

Chicago

Doğan, Ömer, and Burcu Atar. 2026. “DIF Analysis With a Multidimensional Multilevel Mixture Model With Covariates”. International Journal of Educational Studies and Policy, no. Advanced Online Publication. https://doi.org/10.63612/ijesp.1961623.

EndNote

Doğan Ö, Atar B (August 1, 2026) DIF Analysis with a Multidimensional Multilevel Mixture Model with Covariates. International Journal of Educational Studies and Policy Advanced Online Publication

IEEE

[1]Ö. Doğan and B. Atar, “DIF Analysis with a Multidimensional Multilevel Mixture Model with Covariates”, IJESP, no. Advanced Online Publication, Aug. 2026, doi: 10.63612/ijesp.1961623.

ISNAD

Doğan, Ömer - Atar, Burcu. “DIF Analysis With a Multidimensional Multilevel Mixture Model With Covariates”. International Journal of Educational Studies and Policy. Advanced Online Publication (August 1, 2026). https://doi.org/10.63612/ijesp.1961623.

JAMA

1.Doğan Ö, Atar B. DIF Analysis with a Multidimensional Multilevel Mixture Model with Covariates. IJESP. 2026. doi:10.63612/ijesp.1961623.

MLA

Doğan, Ömer, and Burcu Atar. “DIF Analysis With a Multidimensional Multilevel Mixture Model With Covariates”. International Journal of Educational Studies and Policy, no. Advanced Online Publication, Aug. 2026, doi:10.63612/ijesp.1961623.

Vancouver

1.Ömer Doğan, Burcu Atar. DIF Analysis with a Multidimensional Multilevel Mixture Model with Covariates. IJESP. 2026 Aug. 1;(Advanced Online Publication). doi:10.63612/ijesp.1961623