Comparison of Kernel equating methods under NEAT and NEC designs

Year 2023, , 56 - 75, 20.03.2023

Şeyma Nur Özsoy , Sevilay Kilmen

https://doi.org/10.21449/ijate.981367

Abstract

In this study, Kernel test equating methods were compared under NEAT and NEC designs. In NEAT design, Kernel post-stratification and chain equating methods taking into account optimal and large bandwidths were compared. In the NEC design, gender and/or computer/tablet use was considered as a covariate, and Kernel test equating methods were performed by using these covariates and considering bandwidths. The study shows that, in the NEAT design, Kernel chain equating methods exhibit higher error than the post-stratification equating methods do since the lowest error in the NEC design was obtained from the Kernel equating method with large bandwidth through the computer/tablet variable. Kernel test equating results based on the NEC design, which considers gender and computer tablet use variables as a covariate separately, showed lower SEE than that of the NEC pattern, which takes these variables together as covariates. In terms of the bandwidth, when all methods are compared within the pattern used (i.e., NEAT and NEC), it has been seen that generally Kernel test equating with large bandwidth results in fewer errors than the Kernel test equating with optimal bandwidth. When the NEAT and NEC designs are compared generally, the NEAT design has a lower SEE than that of the NEC design.

Keywords

Test equating, Kernel equating, NEAT design, NEC design, Bandwidth

Comparison of Kernel equating methods under NEAT and NEC designs

Abstract

In this study, Kernel test equating methods were compared under NEAT and NEC designs. In NEAT design, Kernel post-stratification and chain equating methods taking into account optimal and large bandwidths were compared. In the NEC design, gender and/or computer/tablet use was considered as a covariate, and Kernel test equating methods were performed by using these covariates and considering bandwidths. The study shows that, in the NEAT design, Kernel chain equating methods exhibit higher error than the post-stratification equating methods do since the lowest error in the NEC design was obtained from the Kernel equating method with large bandwidth through the computer/tablet variable. Kernel test equating results based on the NEC design, which considers gender and computer tablet use variables as a covariate separately, showed lower SEE than that of the NEC pattern, which takes these variables together as covariates. In terms of the bandwidth, when all methods are compared within the pattern used (i.e., NEAT and NEC), it has been seen that generally Kernel test equating with large bandwidth results in fewer errors than the Kernel test equating with optimal bandwidth. When the NEAT and NEC designs are compared generally, the NEAT design has a lower SEE than that of the NEC design.

Keywords

Test equating, Kernel equating, NEAT design, NEC design, Bandwidth

References

• Akın Arıkan, Ç. (2020). The impact of covariate variables on kernel equating under the non-equivalent groups. Eğitimde ve Psikolojide Ölçme ve Değerlendirme, 11(4), 362-373.
• Albano, A.D., & Wiberg, M. (2019). Linking with external covariates: examining accuracy by anchor type, test length, ability difference, and sample size. Applied Psychological Measurement, 43(8), 597-610.
• Andersson, B., Bränberg, K., & Wiberg, M. (2013). Performing the kernel method of test equating with the package kequate. Journal of Statistical Software, 55(6), 1-25.
• Bränberg, K. (2010). Observed score equating with covariates [Unpublished Doctoral dissertation]. Department of Statistics, Umeå University.
• Branberg, K., & Wiberg, M. (2011). Observed score linear equating with covariates. Journal of Educational Measurement, 48(4), 419-440.
• Braun, H.I., & Holland, P.W. (1982). Observed-score test equating: A mathematical analysis of some ETS equating procedures. In P.W. Holland & D.B. Rubin (Eds.) Test equating (9-49). Academic Press.
• Choi, S.I. (2009). A comparison of kernel equating and traditional equipercentile equating methods and the parametric bootstrap methods for estimating standard errors in equipercentile equating (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign.
• Godfrey, K.E. (2007). A comparison of kernel equating and IRT true score equating methods [Unpublished doctoral dissertation]. The Faculty of the Graduate School at the University of North Carolina at Greensboro.
• Gonzales, J., Barrientos, A.F., & Quintana, F.A. (2015). Bayesian nonparametric estimation of test equating functions with covariates. Computational Statistics and Data Analysis, 89, 222-244.
• Gonzales, J., & Wiberg, M. (2017). Applying test equating methods. Springer.
• Kolen, M.J. (1988). Traditional equating methodology. Educational measurement: Issues and practice, 7(4), 29-37.
• Kolen, M.J., & Brennan, R.L. (1995). Test equating: Methods and practises. Springer
• Kolen, M.J., & Brennan, R.L. (2004). Test equating, scaling and linking: Methods and practises. Springer
• Liang, T., & von Davier, A.A. (2014). Cross-validation: An alternative bandwidth-selection method in Kernel equating. Applied Psychological Measurement, 38(4), 281-295.
• Livingston, S.A. (1993). Small-sample equating with log-linear smoothing. Journal of Educational Measurement, 30(1), 23-39.
• Livingston, S.A. (2014). Equating test scores (without IRT). Educational testing service.
• Livingston, S.A., Dorans, N.J., & Wright, N.K. (1990). What combination of sampling equating methods works best?. Applied Measurement in Education, 3, 73–95.
• Lord, F.M. (1980). Applications of item response theory to practical testing problems. Routledge. R Core Team. (2013). R: A language and environment for statistical computing. (Versiyon 4.0.3) [Computer software]. R Foundation for Statistical Computing.
• von Davier, A.A. (2013). Observed-score equatıng: an overvıew. Psychometrika, 78(4), 605-623.
• von Davier, A.A., Holland, P.W., Livingston, S.A., Casabianca, J., Grant, M.C., & Martin, K. (2006). An evaluation of the kernel equating method: A special study with pseudo tests constructed from real test data. ETS Research Report Series, 2006(1), i-31.
• von Davier, A.A., Holland, P.W., & Thayer, D.T. (2004a). The chain and post-stratification methods for observed-score equating: their relationship to population invariance. Journal of Educational Measurement, 41(1), 15-32.
• von Davier, A.A., Holland, P.W., & Thayer, D.T. (2004b). The kernel method of test equating. Springer.
• von Davier, A.A., & Chen, H. (2013). The Kernel levine equipercentile observed‐score equating function. ETS Research Report Series, 2013(2), i-27.
• Wallin G., Häggström J., & Wiberg M. (2018) How to select the bandwidth in kernel equating-An evaluation of five different methods. In Wiberg M., Culpepper S.,Janssen R., González J., & Molenaar D. (Ed.), Quantitative Psychology. IMPS 2017. Springer Proceedings in Mathematics & Statistics, vol 233. Springer. https://doi.org/10.1007/978-3-319-77249-3_8
• Wiberg, M. (2015). A note on equating test scores with covariates. In Ellinor Fackle- Fornius (Ed), Festschrift in Honor of Hans Nyquist on the Occasion of His 65th Birthday. Stockholm University.
• Wiberg, M., & Branberg, K. (2015). Kernel equating under the non-equivalent groups with covariates design. Applied Psychological Measurement, 39(5), 349-361.
• Wiberg, M., & von Davier, A.A. (2017). Examining the impact of covariates on anchor tests to ascertain quality over time in a college admissions test. International Journal of Testing, 17(2), 105-126.
• Yurtçu, M. (2018). The comparison of test equating with covariates using Bayesian nonparametric method (Unpublished master thesis). Hacettepe University, Ankara, Turkey.