Investigation of the Missing Data Imputation Methods on Characteristic Curve Transformation Methods Used in Test Equating

Abstract

In this research, the aim was to evaluate the effect of zero imputation and multiple imputation missing data handling methods on item response theory (IRT) based test equating methods under different conditions. Data in this study was obtained from the administration of the TIMSS 2019 eighth-grade science test. Data sets were formed by randomly selecting a sample of 1000 students with full data from booklets 7 and 8. By deleting data under a completely random missing data mechanism within the scope of common-item nonequivalent groups (CINEG) design, four different data sets were obtained with the missing data rates of 10% or 20% in the new test or in both tests. The missing da ta problem was solved by using zero imputation and multiple imputation methods from these data sets. In this way, 8 different data sets were formed. Then, scaling transformation was performed by using characteristic curve transformation methods (Haebara, Stocking-Lord). Test equating results were reported in terms of observed scores. The root mean square error (RMSE) was used as the evaluation criterion to determine the error involved in test equating. As a result, it was determined that in the case of 10% missing data in both tests, generally lower RMSE values were obtained. It was observed that the multiple imputation method, one of the methods for handling missing data, was the method that produced RMSE values that were both the lowest and closer to the full data set as a reference value compared to the zero-imputation method. In addition, it was determined that, when compared to the Haebara method, Stocking-Lord method, one of the characteristic curve transformation methods, produced lower RMSE values and these values were closer to the full data set, which was taken as a reference value.

Keywords

• Missing data
• zero imputation method
• multiple imputation method
• test equating
• characteristic curve transformation methods

References

1. Akbaş, U. (2014). Farklı örneklem büyüklüklerinde ve kayıp veri örüntülerinde ölçeklerin psikometrik özelliklerinin kayıp veri baş etme teknikleri ile incelenmesi [Investigation of psychometric properties of scales with missing data techniques for different sample sizes and missing data patterns] (Thesis No. 370326) [Doctoral dissertation, Ankara University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
2. Aksekioğlu, B. (2017). Madde tepki kuramına dayalı test eşitleme yöntemlerinin karşılaştırılması: PISA 2012 fen testi örneği [Comparison of test equating methods based on item response theory: PISA 2012 science test sample] (Thesis No. 454879) [Master thesis, Akdeniz University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
3. Allison, P. D. (2003). Missing data techniques for structural equation modeling. Journal of Abnormal Psychology, 112(4), 545-557. https://doi.org/10.1037/0021-843X.112.4.545
4. Angoff, W. H. (1971). Scales, norms and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (pp. 508-600). American Council on Education.
5. Atalay Kabasakal, K. (2014). Değişen madde fonksiyonunun test eşitlemeye etkisi [The effect of differential item functioning on test equating] (Thesis No. 363206) [Doctoral dissertation, Hacettepe University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
6. Aytekin Kazanç, E. (2019). Şans başarısının test eşitlemeye etkisinin farklı eşitleme teknikleriyle araştırılması [Investigation of the effect of guessing on test equating with different equating methods] (Thesis No. 584263) [Master thesis, Hacettepe University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
7. Battauz, M. (2021). Package ‘equateIRT’. https://cran.r-project.org/web/packages/equateIRT/equateIRT.pdf Bayhan, A. (2018). Farklı koşullardaki kayıp veri oranının iç tutarlığa etkisi [The effect of missing data rate on internal consistency within different conditions] (Thesis No. 531022) [Master thesis, Hacettepe University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
8. Bayram, İ. (2020). Kayıp veri ile baş etme yöntemlerinin güvenirlik kestirimleri üzerine etkisi [Comparison of influence of the missing data handling methods on reliability estimation] (Thesis No. 634087) [Master thesis, Hacettepe University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/

9. Büyüköztürk, Ş. (2011). Sosyal bilimler için veri analizi el kitabı [Manual of data analysis for social sciences]. Pegem A Publishing.
10. Chalmers, P., Pritikin, J., Robitzsch, A., Zoltak, M., Kim, K. H., Falk, C. F., Meade, A., Schneider, L., King, D., Liu, C. W., & Ogreten, O. (2021). Package ‘mirt’. https://cran.r-project.org/web/packages/mirt/mirt.pdf
11. Cook, L. L., & Eignor, D. R. (1991). An NCME instructional module on IRT equating methods. Educational Measurement: Issues and Practice, 10(3), 37-45. http://www.edmeasurement.net/8225/Cook-1991-equating-methods.pdf
12. DeMars, C. (2010). Item response theory: Understanding statistics measurement. Oxford Press.
13. Demir, E. (2013). Kayıp verilerin varlığında iki kategorili puanlanan maddelerden oluşan testlerin psikometrik özelliklerinin incelenmesi [Psychometric properties of tests composed of dichotomous items in the presence of missing data] (Thesis No. 342477) [Doctoral dissertation, Ankara University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
14. Demirus, K. B. (2015). Ortak maddelerin değişen madde fonksiyonu gösterip göstermemesi durumunda test eşitlemeye etkisinin farklı yöntemlerle incelenmesi [The study of the effect of anchor items showing or not showing differantial item functioning to test equating using various methods] (Thesis No. 399468) [Doctoral dissertation, Hacettepe University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
15. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates Publishers.
16. Enders, C. K. (2004). The impact of missing data on sample reliability estimates: Implications for reliability reporting practices. Educational and Psychological Measurement, 64(3), 419-436. https://doi.org/10.1177/0013164403261050
17. Enders, C. K. (2010). Applied missing data analysis. The Guildford Press.
18. Ertoprak, D. G. (2017). Kayıp verinin test eşitlemeye etkisinin incelenmesi [Investigating the effect of missing data on test equating] (Thesis No. 470015) [Doctoral dissertation, Hacettepe University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
19. Finch, H. (2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45(3), 225-245. https://doi.org/10.1111/j.1745-3984.2008.00062.x
20. Haebara, T. (1980). Equating logistic ability scales by a weighted least squares method. Japanese Psychological Research, 22(3), 144-149. https://doi.org/10.4992/psycholres1954.22.144
21. Hambleton, R. K., & Jones, RW. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), 38-47. https://doi.org/10.1111/j.1745-3992.1993.tb00543.x
22. Hambleton, R. K., & Swaminathan, H. (1985). Item response theory. Principals and applications. Kluwer Academic Publishers.
23. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Sage Publications, Inc.
24. Han, K. T. (2008). Impact of item parameter drift on test equating and proficiency estimates (UMI No. 3325324) [Doctoral dissertation, University of Massachusetts Amherst]. Available from ProQuest Dissertations and Theses Global database.
25. Harris, D. J., & Crouse, J. D. (1993). A study of criteria used in equating. Applied Measurement in Education, 6(3), 195-240. https://doi.org/10.1207/s15324818ame0603_3
26. Hohensinn, C., & Kubinger, K.D. (2011). On the impact of missing values on the item fit and the model validness of the rasch model. Psychological Test and Assessment Modeling, 53(3), 380-393. https://www.researchgate.net/publication/263655931
27. Işıkoğlu, M. A. (2017). Kayıp veri ile baş etme yöntemlerinin ölçme değişmezliğine etkisi açısından karşılaştırılması [Comparison of influence of the missing data handling methods on measurement invariance] (Thesis No. 484106) [Master thesis, Hacettepe University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
28. Jabrayilov, R., Emons, W. H. M., & Sijtsma, K. (2016). Comparison of classical test theory and item response theory in individual change assessment. Applied Psychological Measurement, 40(8), 559-572. https://doi.org/10.1177/0146621616664046
29. Kan, A. (2011). Test eşitleme: OKS testlerinin istatistiksel eşitliğinin sınanması [Test equating: Checking statistical equivalance of OKS test edition]. Education and Science, 36(160), 38-51. http://eb.ted.org.tr/index.php/EB/article/view/310/258
30. Karagül, A. E. (2020). Küçük örneklemelerde çok kategorili puanlanan maddelerden oluşan testlerde klasik test eşitleme yöntemlerinin karşılaştırılması [Comparison of classical test equating methods with polytomously scored tests and small samples] (Thesis No. 610685) [Master thesis, Ankara University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
31. Karasar, N. (2009). Bilimsel araştırma yöntemi [Scientific research method]. Nobel Publishing.
32. Karkee, T. B., & Wright, K. R. (2004, April 16). Evaluation of linking methods for placing three-parameter logistic item parameter estimates onto a one-parameter scale [Conference presentation]. Annual Meeting of the American Educational Research Association in San Diego, California. https://files.eric.ed.gov/fulltext/ED491663.pdf
33. Keller, L. A., & Keller, R. R. (2011). The long-term sustainability of different Item Response Theory scaling methods. Educational and Psychological Measurement, 71(2), 362-379. https://doi.org/10.1177/0013164410375111
34. Kilmen, S. (2010). Madde Tepki Kuramı’na dayalı test eşitleme yöntemlerinden kestirilen eşitleme hatalarının örneklem büyüklüğü ve yetenek dağılımına göre karşılaştırılması [Comparison of equating errors estimated from test equation methods based on Item Response Theory according to the sample size and ability distribution] (Thesis No. 279926) [Doctoral dissertation, Ankara University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
35. Kim, J. S., & Hanson B. A. (2000). Test equating under the multiple-choice model. Applied Psychological Measurement, 26(3), 255-270. https://doi.org/10.1177/0146621602026003002
36. Kim, M. S. (2015). Linking with planned missing data: Concurrent calibration with multiple imputation [Doctoral dissertation, University of Kansas]. KU Scholar Works. https://kuscholarworks.ku.edu/handle/1808/20985
37. Koçak, D. (2016). Kayıp veriyle baş etme yöntemlerinin madde tepki kuramı bir parametreli lojistik modelinde model veri uyumuna ve standart hataya etkisi [The effect of missing data tecniques in one parameter logistic model of item response theory on model fit and standard error] (Thesis No. 456676) (Doctoral dissertation, Ankara University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
38. Kolen, M. J., & Brennan, R. L. (2004). Test equating, scaling, and linking: Methods and practices (2nd ed.). Springer.
39. Lee, W. C., & Ban, J. C. (2010). A comparison of IRT linking procedures. Applied Measurement in Education, 23(1), 23-48. https://doi.org/10.1080/08957340903423537
40. Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data (2nd ed.). Wiley Publishing.
41. Loyd, B. H., & Hoover, H. D. (1980). Vertical equating using the Rasch model. Journal of Educational Measurement, 17(3), 179-193. https://doi.org/10.1111/j.1745-3984.1980.tb00825.x
42. Marco, G. L. (1977). Item characteristic curve solutions to three intractable testing problems. Journal of Educational Measurement, 14(2), 139-160. https://doi.org/10.1111/j.1745-3984.1977.tb00033.x
43. McKnight, P. E., McKnight, K. M., Sidani, S., & Figueredo, A. J. (2007). Missing data: A gentle introduction. Guilford Press.
44. Mullis, I. V. S., Martin, M. O., Foy, P., Kelly, D. L., & Fishbein, B. (2020). TIMSS 2019 international results in mathematics and science. Boston College, TIMSS & PIRLS International Study Center. https://timssandpirls.bc.edu/timss2019/international-results/
45. Mutluer, C. (2013). Yıl içinde farklı akademik personel ve lisansüstü eğitimi giriş sınavı (ALES) puanlarına ilişkin bir test eşitleme çalışması [A test equating study concerning to ALES (Academic Personnel and Postgraduate Education Entrance Exam) scores obtained at different times in a year] (Thesis No. 336323) [Master thesis, Abant İzzet Baysal University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
46. Mutluer, C. (2021). Klasik Test Kuramı’na ve Madde Tepki Kuramı’na dayalı test eşitleme yöntemlerinin karşılaştırılması: Uluslararası öğrenci değerlendirme programı (PISA) 2012 matematik testi örneği [Comparison of test equating methods based on Classical Test Theory and Item Response Theory: International Student Assessment Program (PISA) 2012 mathematics test case] (Thesis No. 658052) [Doctoral dissertation, Gazi University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
47. Ngudgratoke, S. (2009). An investigation of using collateral information to reduce equating biases of the post-stratification equating method (UMI No. 3381312) [Doctoral dissertation, Michigan State University]. Available from ProQuest Dissertations and Theses Global database.
48. Öztemür, B. (2014). Kayıp veri yöntemlerinin farklı değişkenler altında varyans analizi (t- testi, anova) parametreleri üzerine etkisinin incelenmesi [Examining the effect of missing data methods on variance analysis (t-Test, ANOVA) parameters under different variables] (Thesis No. 357738) [Master thesis, Abant İzzet Baysal University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
49. Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. John Wiley & Sons.
50. Ryan, J., & Brockmann, F. (2009). A practitioner’s introduction to equating with primers on Classical Test Theory and Item Response Theory. CCSSO.
51. Shin, S. H. (2009). How to treat omitted responses in Rasch model-based equating. Practical Assessment Research & Evaluation, 14(1), 1-8. https://doi.org/10.7275/x9vv-xg85
52. Skaggs, G. (2005). Accuracy of random groups equating with very small samples. Journal of Educational Measurement, 42(4), 309-330. https://doi.org/10.1111/j.1745-3984.2005.00018.x
53. Stocking, M. L., & Lord, F. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7(2), 201-210. https://doi.org/10.1177/014662168300700208
54. Swaminathan, J., & Gifford, J. A. (1983). Estimation of parameters in the three-parameter latent trait model. In D. J. Weiss (Ed.), New horizons in testing (pp. 13-30). Academic.
55. Tamcı, P. (2018). Kayıp veriyle başa çıkma yöntemlerinin değişen madde fonksiyonu üzerindeki etkisinin incelenmesi [Investigation of the impact of techniques of handling missing data on differential item functioning] (Thesis No. 517260) [Master thesis, Hacettepe University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
56. Tian, F. (2011). A comparison of equating/linking using the Stocking-Lord method and concurrent calibration with mixed-format tests in the non-equivalent groups commonitem design under IRT [Doctoral dissertation, Boston College]. Boston College Libraries. http://hdl.handle.net/2345/2370
57. Uysal, İ. (2014). Madde Tepki Kuramı’na dayalı test eşitleme yöntemlerinin karma modeller üzerinde karşılaştırılması [Comparison of irt test equating methods for mixed format tests] (Thesis No. 370226) [Master thesis, Abant İzzet Baysal University]. Council of Higher Education Thesis Center. https://tez.yok.gov.tr/UlusalTezMerkezi/
58. Wolf, R. (2013). Assessing the impact of characteristics of the test, common-items, and examinees on the preservation of equity properties in mixed-format test equating [Doctoral dissertation, University of Pittsburgh]. D-Scholarship. http://d-scholarship.pitt.edu/id/eprint/20130
59. Wu, W., Jia, F., & Enders, C. (2015). A comparison of imputation strategies for ordinal missing data on likert scale variables. Multivariate Behavioral Research, 50(5), 484-503. https://doi.org/10.1080/00273171.2015.1022644
60. Yang, W. L. (1997). The effects of content homogeneity and equating method on the accuracy of common-item test equating (UMI No. 9839718) [Doctoral dissertation, Michigan State University]. Available from ProQuest Dissertations and Theses Global database.
61. Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125-145. https://doi.org/10.1177/014662168400800201
62. Zenisky, A. L., Hambleton, R. K., & Sireci, S. G. (2001). Effects of local item dependence on the validity of IRT item, test and ability statistics [Monograph]. MCAT, 3-30. https://eric.ed.gov/?id=ED462426
63. Zhu, X. (2014). Comparison of four methods for handing missing data in longitudinal data analysis through a simulation study. Open Journal of Statistics, 4(11), 933-944. http://dx.doi.org/10.4236/ojs.2014.411088