How Consistent Are Modern DIF Methods? Evidence From an Operational General Education Test

Abstract

Differential item functioning (DIF) analyses contribute evidence relevant to fairness, yet limited empirical work has directly compared newer DIF procedures on operational testing data. This study contrasted three modern DIF methods, Weighted Sum Score (WSS), Robust DIF, and Residual DIF (RDIF), using dichotomously scored data from first-year undergraduates (n = 3,630) who completed a 33-item general education assessment (GLEX4). DIF by sex was evaluated after examining the suitability of a unidimensional 2PL model through item fit, local independence, and unidimensionality checks; two clearly misfitting items were removed prior to DIF analyses. WSS (uniform DIF only), Robust DIF (robust scaling in a common-item design), and RDIF (residual-based indices from pooled calibration) were implemented with false discovery rate control. All three procedures flagged a meaningful set of items for DIF but did not produce identical item classifications. Agreement was strongest for uniform DIF, with the methods converging on a shared subset of items reflecting group differences in difficulty. In contrast, divergence was more pronounced for nonuniform DIF, where RDIF identified more slope-related differences than Robust DIF and WSS was not applicable by design. Overall, findings indicate that these procedures yield partially overlapping DIF profiles and should be treated as complementary rather than interchangeable, with statistical flags interpreted alongside structured substantive item review in applied fairness evaluations.

Keywords

• Differential Item Functioning
• Item Response Theory
• Fairness

References

1. American Educational Research Association, American Psychological Association (APA), & National Council on Measurement in Education (NCME). (Eds.). (2014). Standards for Educational and Psychological Testing. American Educational Research Association.
2. Bandalos, D. L. (2018). Exploratory factor analysis. In Measurement theory and applications for the social sciences (pp. 301–349). Guilford Press.
3. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
4. Christensen, K. B., Makransky, G., & Horton, M. (2017). Critical values for yen’s Q3 identification of local dependence in the rasch model using residual correlations. Applied Psychological Measurement, 41(3), 178–194. https://doi.org/10.1177/0146621616677520
5. Clauser, B. E., & Mazor, K. M. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues and Practice, 17(1), 31–44. https://doi.org/10.1111/j.1745-3992.1998.tb00619.x
6. De Ayala, R. J. (1990). Assessing the fit of dichotomous item response theory models. Behavior Research Methods, Instruments, & Computers, 22(1), 80–81. https://doi.org/10.3758/BF03203121
7. de Ayala, R. J. (2009). The theory and practice of item response theory. Guilford Press.
8. DeMars, C. (2010). Item response theory. Oxford University Press.

9. DeMars, C. (2019). Revised parallel analysis with nonnormal ability and a guessing parameter. Educational and Psychological Measurement, 79(1), 151–169. https://doi.org/10.1177/0013164418767009
10. Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low Kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. https://doi.org/10.1016/0895-4356(90)90158-L
11. General Education Program. (n.d.). American and Global Perspectives: Requirements. Retrieved https://www.jmu.edu/gened/requirements/american-global-perspectives.shtml
12. Guo, H., & Dorans, N. J. (2020). Using weighted sum scores to close the gap between DIF practice and theory. Journal of Educational Measurement, 57(4), 484–510. https://doi.org/10.1111/jedm.12258
13. Guo, W., & Choi, Y.-J. (2023). Assessing dimensionality of IRT models using traditional and revised parallel analyses. Educational and Psychological Measurement, 83(3), 609–629. https://doi.org/10.1177/00131644221111838
14. Gwet, K. L. (2014a). Benchmarking inter-rater reliability coefficients. In Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (Fourth edition, pp. 163–180). Advances Analytics, LLC.
15. Gwet, K. L. (2014b). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (Fourth edition). Advances Analytics, LLC.
16. Halpin, P. (2024). Differential item functioning via robust scaling. Psychometrika, 89(3), 796–821. https://doi.org/10.1007/s11336-024-09957-6
17. Halpin, P. (2025). robustDIF: Differential item functioning using robust statistics (Version 0.2.0) [R]. https://github.com/peterhalpin/robustDIF
18. Holland, P. W., & Thayer, D. T. (1986). Differential item performance and the Mantel–Haenszel procedure. ETS Research Report Series, 1986(2), i–24. https://doi.org/10.1002/j.2330-8516.1986.tb00186.x
19. Kalinowski, S. T. (2019). A Graphical Method for Displaying the Model Fit of Item Response Theory Trace Lines. Educational and Psychological Measurement, 79(6), 1064–1074. https://doi.org/10.1177/0013164419846234
20. Lim, H. (2025). irtQ: Unidimensional Item Response Theory Modeling (Version 1.0.0) [Computer software]. https://CRAN.R-project.org/package=irtQ
21. Lim, H., Choe, E. M., & Han, K. T. (2022). A residual-based differential item functioning detection framework in item response theory. Journal of Educational Measurement, 59(1), 80–104. https://doi.org/10.1111/jedm.12313
22. Lu, R., Guo, H., & Dorans, N. J. (2021). Robustness of weighted differential item functioning analysis: The case of Mantel–Haenszel DIF. ETS Research Report Series, 2021(1), 1–23. https://doi.org/10.1002/ets2.12325
23. Magis, D., Béland, S., Tuerlinckx, F., & De Boeck, P. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862. https://doi.org/10.3758/BRM.42.3.847
24. Myers, N. D., Wolfe, E. W., Feltz, D. L., & Penfield, R. D. (2006). Identifying differential item functioning of rating scale items with the Rasch model: An introduction and an application. Measurement in Physical Education and Exercise Science, 10(4), 215–240. https://doi.org/10.1207/s15327841mpee1004_1
25. Nason, K., & DeMars, C. (2025). Another look at Yen’s Q3: Is .2 an appropriate cut-off? Journal of Educational Measurement, 62(2), 345–359. https://doi.org/10.1111/jedm.12432
26. Orlando, M., & Thissen, D. (2000). Likelihood-based item-fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24(1), 50–64. https://doi.org/10.1177/01466216000241003
27. Penfield, R. D., & Lam, T. C. M. (2000). Assessing differential item functioning in performance assessment: Review and recommendations. Educational Measurement: Issues and Practice, 19(3), 5–15. https://doi.org/10.1111/j.1745-3992.2000.tb00033.x
28. R Core Team. (2024). R: A Language and Environment for Statistical Computing (Version 4.4.2) [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/
29. Rogers, Hj., & Swaminathan, H. (1993). A comparison of logistic regression and Mantel-Haenszel procedures for detecting differential item functioning. Applied Psychological Measurement, 17(2), 105–116. https://doi.org/10.1177/014662169301700201
30. Rubright, J. D., Jodoin, M., Woodward, S., & Barone, M. A. (2022). Differential item functioning analysis of United States Medical Licensing Examination Step 1 items. Academic Medicine, 97(5), 718–722. https://doi.org/10.1097/ACM.0000000000004567
31. Steinberg, L. (2001). The Consequences of Pairing Questions: Context Effects in Personality Measurement. Journal of Personality and Social Psychology, 27(1), 77–83. https://doi.org/10.1037//0022-3514.81.2.332
32. Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x
33. Technical Manual for the Praxis Tests and Related Assessments (p. 53). (2024). [Technical Manual]. Educational Testing Service.
34. Thissen, D., Steinberg, L., & Gerrard, M. (1986). Beyond group-mean differences: The concept of item bias. Psychological Bulletin, 99(1), 118–128. https://doi.org/10.1037/0033-2909.99.1.118
35. Thissen, D., Steinberg, L., & Kuang, D. (2002). Quick and easy implementation of the Benjamini-Hochberg procedure for controlling the false positive rate in multiple comparisons. Journal of Educational and Behavioral Statistics, 27(1), 77–83. https://doi.org/10.3102/10769986027001077
36. Woods, C. M. (2009). Empirical selection of anchors for tests of differential item functioning. Applied Psychological Measurement, 33(1), 42–57. https://doi.org/10.1177/0146621607314044
37. WorkKeys® essential skills technical manual: WorkKeys NCRC assessments technical manual. (2025). Educational Testing Service.
38. Yen, W. M. (1984). Effects of local iItem dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125–145. https://doi.org/10.1177/014662168400800201
39. Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF).
40. Zumbo, B. D. (2007). Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223–233. https://doi.org/10.1080/15434300701375832