An Analysis of Differential Bundle Functioning in Multidimensional Tests Using the SIBTEST Procedure

Year 2022, Volume: 9 Issue: 2, 319 - 336, 26.06.2022

Didem Özdoğan , Hülya Kelecioğlu

https://doi.org/10.21449/ijate.946978

Abstract

This study aims to analyze the differential bundle functioning in multi-dimensional tests with a specific purpose to detect this effect through differentiating the location of the item with DIF in the test, the correlation between the dimensions, the sample size, and the ratio of reference to focal group size. The first 10 items of the test that is comprised of 30 items were acknowledged as the bundle. The data in line with the parameters were generated via SAS program as two categories (1-0) and multidimensional through an extended 2PL model. Differential bundle functioning was detected via the SIBTEST procedure. The results of the study were interpreted according to the criteria of the power rate and the type I error. When the results were reviewed, the analysis of the bundle revealed that the more the correlation between the two dimensions increased, relatively the less the power rates became. It was observed that the power rates, which were obtained according to two different sample sizes in the study, increased as the sample size increased. Another result as to the SIBTEST's power for detecting DIF was the highest when the ratio of reference to focal group size was equal. According to the results of the type I error rate, the error rate was observed to be relatively decreasing as the correlation between the dimensions increased and it was observed to be increasing as the sample size increased. Also, the highest error rate was obtained when the ratio of the samples was equal.

Keywords

Differential item functioning, Differential bundle functioning, Multidimensionality, SIBTEST, Type 1 error, Power rate

References

• Ackerman, T.A. (1992a). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29(1), 67-91. https://doi.org/10.1111/j.1745-3984.1992.tb00368.x
• Ackerman, T.A. (1992b). An investigation of the relationship between reliability, power, and the type I error rate of the Mantel-Haenszel and simultaneous item bias detection procedures. Paper presented at the National Council on Measurement in Education (April 21-23), San Fransisco, CA. https://eric.ed.gov/?id=ED344937
• Ackerman, T.A. (1994). Using multidimensional item response theory to understand what items and tests are measuring. Applied Measurement in Education, 7(4), 255-278. https://doi.org/10.1207/s15324818ame0704_1
• Ackerman, T.A., Gierl, M.J., & Walker, C.M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22(3), 37-51. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
• Atalay Kabasakal K., Arsan N., Gök, B., & Kelecioğlu H. (2014). Değişen madde fonksiyonunun belirlenmesinde mtk olabilirlik oranı sibtest ve mantel-haenszel yöntemlerinin performanslarının (i. tip hata ve güç) karşılaştırılması [Comparing Performances (Type I error and Power) of IRT Likelihood Ratio SIBTEST and Mantel-Haenszel Methods in the Determination of Differential Item Functioning]. Kuram ve Uygulamada Eğitim Bilimleri. 6(14), 2175 2194. https://doi.org/10.12738/estp.2014.6.2165
• Awuor, R.A. (2008). Effect of unequal sample sizes on the power of dif detection: an irt based monte carlo study with SIBTEST and mantel-haenszel procedures. [Doctoral dissertation, Virginia Polytechnic Institute and State University]. https://vtechworks.lib.vt.edu/handle/10919/28321
• Bolt, D.M. (2002). A Monte Carlo comparison of parametric and nonparametric polytomous DIF detection methods. Applied Measurement in Education, 15(2), 113 141. https://doi.org/10.1207/S15324818AME1502_01
• Bolt, D.M., & Lall, V.F. (2003). Estimation of compensatory and noncompensatory multidimensional item response models using Markov Chain Monte Carlo. Applied Psychological Measurement, 27(6), 395 414. https://doi.org/10.1177%2F0146621603258350
• Boughton, K.A., Gierl, M.J., & Khaliq, S.N. (2000). Differential bundle functioning on mathematics and science achievement tests: A small step toward understanding differential performance. Paper presented at the Canadian Society for Studies in Education (May 24 27), Edmonton, Alberta, Canada. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.385.5167&rep=rep1&type=pdf
• Camilli, G. (1992). A conceptual analysis of differential item functioning in terms of a multidimensional item response model. Applied Psychological Measurement, 16(2), 129-147. https://doi.org/10.1177%2F014662169201600203
• Cronbach, L.J. (1990). Essentials of psychological testing (5 ed.). Harper & Row.
• Douglas, J.A., Roussos, L.A., & Stout, W. (1996). Item-bundle DIF hypothesis testing: Identifying suspect bundles and assessing their differential functioning. Journal of Educational Measurement, 33(4), 465 484. https://doi.org/10.1111/j.1745 3984.1996.tb00502.x
• Engelhard, G., Hansche, L., & Rutledge, K.E. (1990). Accuracy of bias review judges in identifying differential item functioning on teacher certification tests. Applied Measurement in Education, 3(4), 347 360. https://doi.org/10.1207/s15324818ame0304_4
• Finch, W.H. (2012). The MIMIC model as a tool for differential bundle functioning detection. Applied Psychological Measurement, 36(1), 40 59. https://doi.org/10.1177%2F0146621611432863
• Gierl, M.J., Bisanz, J., Bisanz, G.L., Boughton, K.A., & Khaliq, S.N. (2001). Illustrating the utility of differential bundle functioning analysis to identify and interpret group differences on achievement tests. Educational Measurement: Issues and Practice, 20(2), 26-36. https://doi.org/10.1111/j.1745-3992.2001.tb00060.x
• Harwell, M., Stone, C.A., Hsu, T.C., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20(2), 101 125. https://doi.org/10.1177%2F014662169602000201
• Mahmood UI.H., & Frank, M. (2020). Discrimination with unidimensional and multidimensional item response theory models for educational data. Communications in Statistics Simulatıon and Computation. 1 21. https://doi.org/10.1080/03610918.2019.1705344
• Karasar, N. (2020). Bilimsel araştırma yöntemi, Kavramlar İlkeler (35. Baskı) Teknikler [Scientific Research Method, Concepts Principles Techniques (35 ed.)]. Nobel Yayıncılık.
• Lee, Y. (2004). The impact of a multidimensional item on differential item functioning (DIF). [Doctoral dissertation, University of Washington]. https://www.proquest.com/openview/2e24c73698bf27f10d35bd8b63e2cc31/1?pq-origsite=gscholar&cbl=18750&diss=y
• Nandakumar, R. (1993). Simultaneous DIF amplification and cancellation: Shealy-Stout’s test for DIF. Journal of Educational Measurement, 30(4), 293 311. https://doi.org/10.1111/j.1745-3984.1993.tb00428.x
• Narayanan, P., & Swaminathan, H. (1994). Performance of the Mantel-Haenszel and simultaneous item bias procedures for detecting differential item functioning. Applied Psychological Measurement, 18(4), 315 328. https://doi.org/10.1177%2F014662169401800403
• Oshima, T.C., & Miller, M.D. (1992). Multidimensionality and item bias in item response theory. Applied Psychological Measurement, 16(3), 237 248. https://doi.org/10.1177%2F014662169201600304
• Reckase, M.D., & McKinley, R.L. (1991). The discrimination power of items that measure more than one dimension. Applied Psychological Measurement, 15(4), 361-373. https://doi.org/10.1177%2F014662169101500407
• Ross, T.R. (2008). The impact of multidimensionality on the detection of differential bundle functioning using simultaneous item bias test [Doctoral dissertation, Georgia State University]. https://scholarworks.gsu.edu/eps_diss/14/
• Roussos, L., & Stout, W. (1996). A multidimensionality-based DIF analysis paradigm. Applied Psychological Measurement, 20(4), 355 371. https://doi.org/10.1177%2F014662169602000404
• Russell, S.S. (2005). Estimates of type I error and power for indices of differential bundle and test functioning [Doctoral dissertation, Graduate College of Bowling Green State University]. https://www.proquest.com/openview/25873a6f54d69f576b5c2d3ac61f3aa3/1?pq-origsite=gscholar&cbl=18750&diss=y
• Shealy, R., & Stout, W. (1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58, 159-194.
• Shepard, L.A., Camilli, G., & Averill, M. (1981). Comparison of procedures for detecting test-item bias with both internal and external ability criteria. Journal of Educational Statistics, 6, 317-375. https://doi.org/10.2307/1164616
• Wiley, D.E. (1991). Test validity and invalidity reconsidered. In R. Snow & D.E. Wiley (Eds.), Improving inquiry in social science: a volume in honor of Lee J. Cronbach. Routledge.
• Yao, L., & Boughton, K.A. (2007). A multidimensional item response modeling approach for improving subscale proficiency estimation and classification. Applied Psychological Measurement, 31(2), 83-105. https://doi.org/10.1177%2F0146621606291559