The Impact of Covariate Variables on Kernel Equating under the Non-equivalent Groups

Abstract

This study aims to use covariate variables correlated with the test scores instead of common items for non-equivalent groups with covariates (NEC) design in kernel equating. This study used the 2016 Monitoring and Evaluation of Academic Skills Project in Turkey. The study used data from 6,000 students, randomly selected from the Turkish Ministry of National Education’s current student data. Three thousand of the students took form A, and 3,000 of them took form B. The data include mathematics test scores and consist of 18 items, nine of which are the first items, and nine of which are anchor items. The equated scores from the NEC design were compared with equated scores from the non-equivalent group (NEAT) design. From the equating results, the root mean squared difference (RMSD) and standard error of equating (SEE) values were calculated. The results showed that NEC design could produce lower standard errors compared with the NEAT design, and the least RMSD was provided by NEAT PSE methods and NEC methods. The general result of this research is that test forms can be equated using covariates when there are no anchor items.

Keywords

• NEC design
• NEAT design
• covariate variables
• SEE
• RMSD

References

1. Akın-Arıkan, Ç., & Gelbal, S. (2018). A comparison of traditional and kernel equating methods. International Journal of Assessment Tools in Education, 5(3), 417-427. doi: 10.21449/ijate.409826
2. Albano, A. D., & Wiberg, M. (2019). Linking with external covariates: examining accuracy by anchor type, test length, ability difference, and sample size. Applied psychological measurement, 43(8), 597-610. doi: 10.1177/0146621618824855
3. Andersson, B., & Wiberg, M. (2017). Item response theory observed-score kernel equating. Psychometrica, 82(1), 48-66. doi: 10.1007/s11336-016-9528-7
4. Andersson, B., Branberg, K., & Wiberg, M. (2013). Performing the kernel method of test equating with the package kequate. Journal of Statistical Software, 55(6), 1-25. Retrieved from https://www.jstatsoft.org/article/view/v055i06
5. Branberg, B. (2010). Observed score equating with covariates (Statistical Studies No. 41). Umea: Umea Unıversity, Department of Statistics. Retrieved from https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A306427&dswid=8645
6. Branberg, K., & Wiberg, M. (2011). Observed score linear equating with covariates. Journal of Educational Measurement, 48(4), 419-440. doi: 10.1111/j.1745-3984.2011.00153.x
7. Chen, H., & Holland, P. (2010). New equating methods and their relationships with Levine observed score linear equating under the kernel equating framework. Psychometrika, 75(3), 542-557. doi: 10.1007/S11336-010-9171-7
8. Choi, S. I. (2009). A comparison of kernel equating and traditional equipercentile equating methods and the parametric bootstrap methods for estimating standard errors in equipercentile equating (Unpublished doctoral thesis). University of Illinois at Urbana-Champaign.

9. Cook, L. L., Eignor, D. R., & Schmitt, A. P. (1990). Equating achievement tests using samples matched on ability (College Board Report No. 90-2). New York: College Entrance Examination Board.
10. Gonzalez, J., Barrientos, A. F., & Quintana, F. A. (2015). Bayesian non-parametric estimation of test equating functions with covariates. Computational Statistics and Data Analysis, 89, 222-244. doi: 10.1016/j.csda.2015.03.012
11. Holland, P. W., Dorans, N. J., & Petersen, N. S. (2007). Equating test scores. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics (Vol. 26, pp. 169-203). Oxford, UK: Elsevier.
12. Holland, P. W., & Thayer, D. T. (1989). The kernel method of equating score distributions (ETS RR-89-07). Princeton NJ: ETS.
13. Kolen, M. J. (1990). Does matching in equating work? A discussion. Applied Measurement in Education, 3(1), 97-104.
14. Kolen, M. J., & Brennan, R. L. (2014). Test equating, scaling, and linking: Methods and practices (3nd. ed.). New York: Springer.
15. Liou, M., Cheng, P. E., & Johnson, E. G. (1997). Standard errors of the kernel equating methods under the common-item design. Applied Psychological Measurement, 21(4), 349-369.
16. Liou, M., Cheng, P. E., & Li, M. (2001). Estimating comparable scores using surrogate variables. Applied Psychological Measurement, 25(2), 197-207. doi: 10.1177/01466210122032000
17. Livingston, S. A., Dorans, N. J., & Wright, N. K. (1990). What combination of sampling and equating methods works best? Applied Measurement in Education, 3(1), 73-95.
18. Mao, X. (2006). An investigation of the accuracy of the estimates of standard errors for the Kernel equating functions. (Unpublished doctoral thesis). University of Iowa, Iowa City.
19. MoNE, Measurement Assessment and Examination Services General Directorate [Milli Eğitim Bakanlığı Ölçme Değerlendirme ve Sınav Hizmetleri Genel Müdürlüğü] (2016). Monitoring and evaluating of academic skills study, 8th students report (ABIDE) [Akademik becerilerin izlenmesi ve değerlendirilmesi, 8. sınıflar raporu]. Ankara: Republic of Turkey Ministry of National Education.
20. R Core Team. (2013). R: A language and environment for statistical /computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/.
21. Sansivieri, V., & Wiberg, M. (2016). IRT observed-score equating with the nonequivalent groups with covariates design. In L. A. van der Ark, M. Wiberg, S. A. Culpepper, J. A. Douglas, & W.C. Wang (Eds.), Quantitative psychology- 81st annual meeting of the psychometric society (pp. 275-285). Asheville, NC: Springer.
22. von Davier, A. A., Fournier‐Zajac, S., & Holland, P. W. (2007). An equipercentile version of the Levine linear observed‐score equating function using the methods of kernel equating. (Research Report No: RR-87-31). Princeton, NJ: Educational Testing Service.
23. von Davier, A. A., Holland, P. W., & Thayer, D. T. (2004). The kernel method of test equating. New York: Springer Verlag.
24. von Davier, A. A., Holland, P. W., Livingston, S. A., Casabianca, J., Grant, M. C., & Martin, K. (2006). An evaluation of the Kernel equating method. A special study with pseudotests constructed from real test data (Research Report No: RR-06-02). Princeton, NJ: Educational Testing Service.
25. Wallin, G., & Wiberg, M. (2017). Non-equivalent groups with covariates design using propensity scores for kernel equating. In L. A. van der Ark, M. Wiberg, S. A. Culpepper, J. A. Douglas, & W. C. Wang (Eds.), Quantitative psychology – 81st annual meeting of the psychometric society (pp. 309-319). Asheville, NC: Springer.
26. Wiberg, M. (2015). Anote on equating test scores with covariates. In E. Frackle-Fornius (Ed.), Festschrift in honor of Hans Nyquist on the occasion of his 65th birthday (pp. 96-99). Stockholm, Sweden: Department of Statistics, Stockholm University.
27. Wiberg, M., & von Davier, A. A. (2017). Examining the impact of covariates on anchor tests to ascertain quality over time in a college admissions test, International Journal of Testing, 17(2), 105-126. doi: 10.1080/15305058.2016.1277357
28. Wiberg. M., & Branberg, K. (2015). Kernel equating under the non-equivalent groups with covariates design. Applied Psychological Measurement, 39(5), 349-361. doi: 10.1177/0146621614567939
29. Yurtçu, M. (2018). Parametrik olmayan bayes yöntemiyle ortak değişkenlere göre yapilan test eşitlemelerinin karşilaştirilmasi (Yayınlanmamış doktora tezi). Hacettepe Üniversitesi, Eğitim Bilimleri Enstitüsü, Ankara.