Comparison of the methods of examining measurement equivalence under different conditions in terms of statistical power ratios

Abstract

Validity is the most important psychometric feature that should be found in a measurement tool. Measurement equivalence is one of the evidences for the validity of measurement tools. Providing information to the researchers about the methods of identifying measurement equivalence may contribute to present a complete validity evidence. The purpose of this study is to compare the statistical power ratios of methods used for examining the measurement equivalence based on structural equation modeling and item response theory on the artificial data sets generated by diversifying variables of sample size, number of items, and the ratios of the items having differential item function. In accordance with this purpose, the variables have been varied to include three different levels. In the analysis, multi-group confirmatory factor analysis was used which is among the methods based on the structural equation modeling, and likelihood ratio test and comparison of the item parameters methods were used which are among the methods based on item response theory. Multi-group group confirmatory factor analysis (93,50%) and likelihood ratio test (96,75%) methods have reached the highest statistical power ratio in the condition that the sample size is 1000/1000, the number of items is 40 items and the ratios of the items having differential item function is 10%. The method of comparing the item parameters (94,50%) has reached the highest statistical power ratio in the condition of sample size is 1000/1000, the number of items is 20 and the ratios of the items having differential item function are 10-20%.

Keywords

• Measurement equivalence
• statistical power ratios
• multi-group confirmatory factor analysis
• likelihood ratio test
• comparison of the item parameters method

Ölçme eşdeğerliğini incelemede kullanılan yöntemlerin farklı koşullar altında istatistiksel güç oranları açısından karşılaştırılması

Abstract

Geçerlilik, bir ölçme aracında bulunması gereken en önemli psikometrik özelliktir. Ölçme eşdeğerliği, ölçme araçlarının geçerliliğinin kanıtlarından biridir. Araştırmacılara ölçme eşdeğerliğini belirleme yöntemleri hakkında bilgi verilmesi, tam bir geçerlilik kanıtı sunulmasına katkı sağlayabilir. Bu çalışmanın amacı, yapısal eşitlik modellemesi ve madde tepki kuramına dayalı ölçme eşdeğerliğini incelemek için kullanılan yöntemlerin istatistiksel güç oranlarını, örneklem büyüklüğü, madde sayısı ve değişen madde fonksiyonu içeren madde oranı değişkenleri değişimlenerek oluşturulan yapay veri setleri üzerinde karşılaştırmaktır. Bu amaç doğrultusunda değişkenler üç farklı düzeyi olacak şekilde değişimlenmiştir. Analizde yapısal eşitlik modellemesine dayalı yöntemlerden çok gruplu doğrulayıcı faktör analizi, madde yanıt teorisine dayalı yöntemlerden olabilirlik oranı testi ve madde parametrelerinin karşılaştırılması yöntemleri kullanılmıştır. Örneklem büyüklüğü 1000/1000, madde sayısının 40 olduğu ve değişen madde fonksiyonuna sahip maddelerin oranları ise %10 koşulda çok gruplu grup doğrulayıcı faktör analizi (%93,50) ve olabilirlik oran testi (%96,75) yöntemleri en yüksek istatistiksel güç oranına ulaşmıştır. Madde parametrelerinin karşılaştırılması yöntemi (%94,50) örneklem büyüklüğünün 1000/1000 olması, madde sayısının 20 olması ve diferansiyel madde fonksiyonuna sahip maddelerin oranlarının 10-20% olması durumunda en yüksek istatistiksel güce ulaşmıştır.

Keywords

• Ölçüm eşdeğerliği
• istatistiksel güç oranları
• çoklu grup doğrulayıcı faktör analizi
• olabilirlik oranı testi
• madde parametrelerinin karşılaştırılması yöntemi

References

1. Angoff, W.H. (1993). Perspectives On Differential Item Functioning Methodology. Holland and Wainer (Ed.), Differential Item Functioning içinde (s. 3-23). Lawrence Erlbaum Associates, Publishers, New Jersey.
2. Ankenmann, R. D., Witt, E. A., & Dunbar, S. B. (1999). An Investigation of the Power of the Likelihood Ratio Goodness‐of‐Fit Statistic in Detecting Differential Item Functioning. Journal of Educational Measurement, 36(4), 277-300. Retrieved from https://doi.org/10.1111/j.1745-3984.1999.tb00558.x
3. Atalay, K., Gök, B., Kelecioğlu, H., & Arsan, N. (2012). Değişen madde fonksiyonunun belirlenmesinde kullanılan farklı yöntemlerin karşılaştırılması: Bir simülasyon araştırması. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi, 43, 270-281.
4. Atar, B., & Kamata, A. (2011). Comparison of IRT likelihood ratio test and logistic regression DIF detection procedures. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi, 41(41).
5. Bock, R. D., Murakl, E., & Pfeiffenberger, W. (1988). Item pool maintenance in the presence of item parameter drift. Journal of Educational Measurement, 25(4), 275-285.
6. Bolt, D. M. (2002). A Monte Carlo Comparison of Parametric and Nonparametric Polytomous DIF detection methods. Applied Measurement in Education, 15,113-141. Retrieved from https://doi.org/10.1207/S15324818AME1502_01
7. Clauser, B. E., & Mazor, K. M. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement: issues and practice, 17(1), 31-44. Retrieved from https://doi.org/10.1111/j.1745-3992.1998.tb00619.x
8. Dodeen, H. (2004). Stability of differential item functioning over a single population in survey data. The Journal of experimental education, 72(3), 181-193. Retrieved from https://doi.org/10.3200/JEXE.72.3.181-193

9. Drasgow, F., & Kanfer, R. (1985). Equivalence of psychological measurement in heterogeneous populations. Journal of Applied Psychology, 70(4), 662. Retrieved from http://dx.doi.org/10.1037/0021-9010.70.4.662
10. Elosua, P., & Wells, C. S. (2013). Detecting DIF in polytomous items using MACS, IRT and ordinal logistic regression. Psicológica, 34(2)
11. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.
12. Fidalgo, A. M., Mellenbergh G. J., & Muniz, J. (2000). Effects of Amount of DIF, Test Length, and Purification Type on Robustness and Power of Mantel-Haenszel Procedures. Methods of Psychological Research Online. 5(3), 43-53.
13. Flowers, C.P., Raju, N.S., & Oshima, T. C. (2002). A comparison of measurement equivalence methods based on confirmatory factor analysis and item response theory. Paper presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans.
14. Galton, F. (1884). Measurement of character. Fortnightly Review, 36, 179-185.
15. González Roma, V., Hernandez, A., & Gomez-Benito, J. (2006). Power and Type I error of the mean and covariance structure analysis model for detecting differential item functioning in graded response items. Multivariate Behavioral Research, 41(1), 29-53. Retrieved from https://www.tandfonline.com/doi/abs/10.1207/s15327906mbr4101_3
16. Güngör Culha, D. (2012). Örtük sınıf analizlerinde ölçme eşdeğerliğinin incelenmesi. [Investigating measurement equivalence with latent class analysis] (Yayımlanmamış doktora tezi). Ege Üniversitesi, İzmir.
17. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of ıtem response theory. CA: Sage.
18. Holmes Finch, W., & French, B. F. (2007). Detection of crossing differential item functioning: A comparison of four methods. Educational and psychological Measurement, 67(4), 565-582. Retrieved from https://doi.org/10.1177/0013164406296975
19. Hong, S., Malik, M.L., & Lee, M. (2003). Testing configural, metric, scalar and latent mean invariance across genders in sociotropy and autonomy using a non- western sample. Educational and Psychological Measurement, 63(4), 636-654. Retrieved from https://doi.org/10.1177/0013164403251332
20. Huang, C. D., Church, A. T., & Katigbak, M. S. (1997). Identifying cultural differences in items and traits: Differential item functioning in the NEO Personality Inventory. Journal of Cross-Cultural Psychology, 28, 192-218. Retrieved from http://dx.doi.org/10.1177/0022022197282004
21. Kankaras, M., Vermunt, J. K., & Moors, G. (2011). Measurement equivalence of ordinal items: A comparison of factor analytic, item response theory, and latent class approaches. Sociological Methods & Research, 40(2), 279-310. Retrieved from https://doi.org/10.1177/0049124111405301
22. Karasar, N. (2005). Bilimsel araştırma yöntemi [Scientific research method]. Ankara: Nobel yayın dağıtım
23. Kazelskis, R., Thames, D., & Reeves, C. (2004). The elementary reading attitude survey: factor invariance across gender and race. Reading Psychology, 25, 111-120. Retrieved from https://doi.org/10.1080/02702710490435682
24. Kim, S. H., & Cohen, A. S. (1998). Detection of differential item functioning under the graded response model with the likelihood ratio test. Applied Psychological Measurement, 22(4), 345-355. Retrieved from https://doi.org/10.1177/014662169802200403
25. Kim, E. S., & Yoon, M. (2011). Testing measurement invariance: A comparison of multiple-group categorical CFA and IRT. Structural Equation Modeling, 18(2), 212-228. Retrieved from https://doi.org/10.1080/10705511.2011.557337
26. Korkmaz, M. (2005). Madde cevap kuramına dayalı olarak çok kategorili maddelerde madde ve test yanlılığının (işlevsel farklılığın) incelenmesi. [An investigation of differential item and test functioning at polytomous items based on item response theory]. (Doktora tezi) Ege Üniversitesi, İzmir.
27. Little, T. D. (1997). Mean and covariance structures (MACS) analyses of cross-cultural data: Practical and theoretical issues. Multivariate Behavioral Research, 32(1), 53-76. Retrieved from https://doi.org/10.1207/s15327906mbr3201_3
28. Meade, A. W., & Lautenschlager, G. J. (2004). A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods, 7(4), 361-388. Retrieved from https://doi.org/10.1177/1094428104268027
29. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525-543. Retrieved from https://doi.org/10.1007/BF02294825
30. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
31. Morales, L. S., Reise, S. P., & Hays, R. D. (2000). Evaluating the equivalence of health care ratings by whites and Hispanics. Medical care, 38(5), 517.
32. Muraki, E., & Bock, R. D. (1991). PARSCALE: Parameter scaling of rating data. Chicago: Scientific Software, Inc.
33. Nachtigall, C., Kroehne, U., Funke, F., & Steyer, R. (2003). (Why) should we use SEM? Pros and Cons of Structural Equation Modeling. Methods of Psychological Research Online, 8(2), 1- 22.
34. Narayanan, P., & Swaminathan, H. (1994). Performance of the Mantel-Haenszel and simultaneous item bias procedures for detecting differential item functioning. Applied Psychological Measurement, 18(4), 315-328. Retrieved from https://doi.org/10.1177/014662169401800403
35. Reise, S. P., Smith, L., & Furr, R. M. (2001). Invariance on the NEO PI-R neuroticism scale. Multivariate Behavioral Research, 36(1), 83-110. Retrieved from https://doi.org/10.1207/S15327906MBR3601_04
36. Reise, S. P., Widaman, K. F., & Pugh, R. H. (1993). Confirmatory factor analysis and item response theory: two approaches for exploring measurement invariance. Psychological Bulletin, 114(3), 552-566. Retrieved from https://doi.org/10.1037/0033-2909.114.3.552
37. Rogers, J. H., & Swaminathan, H. (1993). A comparison of logistic regression and mantel-haenszel procedures for detecting differential ıtem functioning. Applied Psychological Measurement, 17(2), 105-116. Retrieved from https://doi.org/10.1177/014662169301700201
38. Somer, O. (2004). Gruplararası karşılaştırmalarda ölçek eşdeğerliğinin incelenmesi: Madde ve test fonksiyonlarının farklılaşması. Türk Psikoloji Dergisi, 19(53), 69 - 82.
39. Somer, O., Korkmaz, M., Dural, S., & Can, S. (2009). Ölçme eşdeğerliğinin yapısal eşitlik modellemesi ve madde tepki kuramı kapsamında incelenmesi. [Detection of Measurement Equivalence by Structural Equation Modeling and Item Response Theory]. Türk Psikoloji Dergisi, 24(64), 61-75.
40. Stark, S., Chernyshenko, O. S., & Drasgow, F. (2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91(6), 1292. Retrieved from https://doi.org/10.1037/0021-9010.91.6.1292
41. Tekin, H. (1991). Eğitimde ölçme ve değerlendirme. [Measurement and evaluation in education]. (6. baskı). Ankara: Yargı Yayınları.
42. Thissen, D. (1991). MULTILOG: Multiple category item analysis and test scoring using item response theory. Chicago, IL: Scientific Software International.
43. Thissen, D., Steinberg, L., & Gerrard, M. (1986). Beyond group-mean differences: The concept of item bias. Psychological Bulletin, 99(1), 118. Retrieved from http://dx.doi.org/10.1037/0033-2909.99.1.118
44. Thissen, D., Steinberg, L., & Wainer, H. (1993). Detection of differential item functioning using the parameters of item response models. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 67-113). Hillsdale, NJ, US: Lawrence Erlbaum Associates, Inc.
45. Van de Vijver, F. J. (1998). Towards a theory bias and equivalence. Zuma Nachrichten: Cross-Cultural Survey Equivalence, 3, 41-65. Retrieved from https://nbn-resolving.org/urn:nbn:de:0168-ssoar-49731-1
46. Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-69. Retrieved from https://doi.org/10.1177/109442810031002
47. Wang, W. C., & Yeh, L. Y. (2003). Effects of anchor item methods on differential item functioning detection with the likelihood ratio test. Applied Psychological Measurement, 27, 479-498. Retrieved from https://doi.org/10.1177/0146621603259902
48. Widaman, K. F., & Reise, S. P. (1997). Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. The science of prevention: Methodological advances from alcohol and substance abuse research, 281-324. Retrieved from http://dx.doi.org/10.1037/10222-009
49. Wu, A. D., Li, Z., & Zumbo, B. D. (2007) Decoding the meaning of factorial invariance and updating the practice of multiple-group confirmatory factor analysis: A demonstration with TIMSS data. Practical Assessment, Research and Evaluation, 12, 1-26.