Covariate-Based Approaches in Kernel Equating: A Comparison with the Anchor-Based Design

Abstract

Test equating plays an important role in ensuring the comparability of scores obtained from different test forms. Traditional approaches rely on common items to adjust for differences in ability between groups or variations in difficulty across forms. However, due to functional, security-related and test design constraints associated with the use of common items alternative covariate-based equating approaches have been developed. The primary aim of this study is to compare the equating performance of anchor-based and covariate-based approaches using real assessment data. Data from the Programme for International Student Assessment were used and five background variables with strong predictive power for reading literacy were selected. Within the anchor-based design, post-stratification and chained equating methods were applied, whereas the covariate-based design employed approaches based on categorized continuous covariates, pseudo-equivalent groups and propensity scores. All analyses were conducted within the framework of Classical Test Theory using observed-score kernel equating. Equating performance was evaluated using the standard error of equating. Results indicated that under the anchor-based design post-stratification equating outperformed chained equating. When designs were compared, covariate-based approaches generally produced lower equating errors than chained equating under the anchor-based design. Only the propensity score approach yielded higher errors than post-stratification equating. The approach using raw covariate scores demonstrated clearly superior performance compared to all other methods. These findings suggest that in addition to common items, background variables can serve as a viable alternative in test equating studies.

Keywords

• test equating
• covariate-based equating
• kernel equating

Test Eşitlemede Ortak Test Desenine Karşı Ortak Değişken Deseni Yaklaşımları

Abstract

Test eşitleme, farklı test formlarından elde edilen puanların karşılaştırılabilirliğinin sağlanmasında önemli bir rol oynamaktadır. Geleneksel yaklaşımlar, gruplar arasındaki yetenek farklılıklarını veya formlar arasındaki güçlük farklılıklarını düzenlemek amacıyla ortak maddelere dayanmaktadır. Ancak ortak maddelerin kullanımına ilişkin bir takım sınırlılıklar kaynaklı ortak değişken temelli alternatif eşitleme yaklaşımları geliştirilmiştir. Bu çalışmanın temel amacı, gerçek değerlendirme verileri kullanarak ortak madde temelli ve ortak değişken temelli yaklaşımların eşitleme performanslarını karşılaştırmaktır. Çalışmada Uluslararası Öğrenci Değerlendirme Programı (PISA) verileri kullanılmış ve okuma becerilerini güçlü biçimde yordayan beş ortak değişken seçilmiştir. Ortak madde temelli desende son-tabakalama ve zincir eşitleme yöntemleri uygulanırken, ortak değişken temelli desende sürekli değişkenlerin kategorik hâle getirilmesine dayalı ham puan yaklaşımı, sözde eşdeğer gruplar ve eğilim puanı yaklaşımları kullanılmıştır. Tüm analizler Klasik Test Kuramı çerçevesinde gözlenen puanlara dayalı Kernel eşitleme yöntemi ile gerçekleştirilmiştir. Eşitleme performansı, eşitlemenin standart hatası (SEE) kullanılarak değerlendirilmiştir. Bulgular, ortak madde temelli desende son-tabakalama eşitlemenin zincir eşitlemeye göre daha iyi performans gösterdiğini ortaya koymuştur. Desenler karşılaştırıldığında, ortak değişken temelli yaklaşımların genel olarak ortak madde temelli desendeki zincir eşitlemeye göre daha düşük eşitleme hataları ürettiği görülmüştür. Yalnızca eğilim puanı yaklaşımı, son-tabakalama eşitlemeye kıyasla daha yüksek hata değerleri üretmiştir. Ham ortak değişken puanlarına dayalı yaklaşım ise diğer tüm yöntemlere kıyasla açık biçimde üstün performans göstermiştir. Bu bulgular, ortak maddelere ek olarak arka plan değişkenlerinin de test eşitleme çalışmalarında uygulanabilir bir alternatif oluşturabileceğini göstermektedir.

Keywords

• test eşitleme
• ortak değişken temelli test eşitleme
• kernel eşitleme

References

1. Akın Arıkan, C. (2020). The Impact of Covariate Variables on Kernel Equating under the Non-equivalent Groups. Journal of Measurement and Evaluation in Education and Psychology, 11(4), 362–373. https://doi.org/10.21031/epod.706835
2. Altintas, O., & Wallin, G. (2021). Equality of admission tests using kernel equating under the non-equivalent groups with covariates design. 8(4), 729–743.
3. Austin, P. C. (2008). Goodness-of-fit diagnostics for the propensity score model when estimating treatment effects using covariate adjustment with the propensity score y. October, 1202–1217. https://doi.org/10.1002/pds
4. Barth, R. P., Shenyang Guo, & McCrae, J. S. (2008). Propensity score matching strategies for evaluating the success of child and family service programs. Research on Social Work Practice, 18(3), 212–222. https://doi.org/10.1177/1049731507307791
5. Sezer Başaran, E., Mutluer, C., & Çakan, M. (2023). A Comparison of Covariates, Equating Designs, and Methods in Equating TIMSS 2019 Science Tests. Participatory Educational Research, 10(5), 41-63. https://doi.org/10.17275/per.23.74.10.5
6. Batista, G. E. A. P. A., & Monard, M. C. (2002). A study of k-nearest neighbour as an imputation method. Frontiers in Artificial Intelligence and Applications, 87, 251–260.
7. Bränberg, K., & Wiberg, M. (2011). Observed score linear equating with covariates. Journal of Educational Measurement, 48(4), 419–440. https://doi.org/10.1111/j.1745-3984.2011.00153.x
8. Budescu, D. (1985). Efficiency of linear equating as a function of the length of the anchor test. Journal of Educational Measurement, 22(1), 13–20. https://doi.org/10.1111/j.1745-3984.1985.tb01045.x

9. Doğaç, A. (2021). PISA 2018 okuma becerilerini açıklayan değişkenlerin çok düzeyli yapısal eşitlik modeli ile incelenmesi (Master’s thesis, Hacettepe University).
10. Dorans, N. J., Pommerich, M., & Holland, P. W. (2007). Linking and aligning scores and scales. In Nonparametric Statistics for Social and Behavioral Sciences. https://doi.org/10.1201/b16188
11. González, J., & Wiberg, M. (2017). Applying test equating methods using R. Springer International Publishing.
12. Haberman, S. J. (1984). Adjustment by Minimum Discriminant Information. Institute of Mathematical Statistics, 12(3), 971–988.
13. Haberman, S. J. (2015). Pseudo-Equivalent Groups and Linking. Journal of Educational and Behavioral Statistics, 40(3), 254–273. https://doi.org/10.3102/1076998615574772
14. Holland, P. W., & Dorans, N. J. (2006). Linking and equating. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 189–220). Praeger Publishers.
15. Holland, P. W., Dorans, N. J., & Petersen, N. S. (2006). 6 equating test scores. In C. R. Rao & S. Sinharay (Eds.), Handbook of Statistics (pp. 169–203). Elsevier. https://doi.org/10.1016/S0169-7161(06)26006-1
16. Kim, S., & Lu, R. (2018). The pseudo‐equivalent groups approach as an alternative to common‐item equating. ETS Research Report Series, 2018(1), 1-13.
17. Kolen, M. J., & Brennan, R. L. (2004). Test equating, scaling, and linking: Methods and practices (2nd ed.). Springer.
18. Kolen, M. J., & Brennan, R. L. (2014). Test equating, scaling, and linking: Methods and practices (3rd ed.). Springer.
19. Laukaityte, I., Wallin, G., & Wiberg, M. (2025). Combining Propensity Scores and Common Items for Test Score Equating. 0(0), 1–30. https://doi.org/10.1177/01466216251363240
20. Lee, J. (2020). Non-Cognitive Characteristics and Academic Achievement in Southeast Asian Countries Based on PISA 2009, 2012, and 2015. OECD Education Working Papers, No. 233. OECD Education Working Paper, 233(233), 1–80. https://dx.doi.org/10.1787/c3626e2f-en
21. Livingston, S. A. (2014). Equating Test Scores (without IRT). Educational Testing Service.
22. Liou, M., Cheng, P. E., & Li, M. Y. (2001). Estimating comparable scores using surrogate variables. Applied Psychological Measurement, 25(2), 197–207. https://doi.org/10.1177/01466210122032000
23. Longford, N. T. (2015). Equating without an anchor for nonequivalent groups of examinees. Journal of Educational and Behavioral Statistics, 40(3), 227–253. https://doi.org/10.3102/1076998615574773
24. Lord, F. M. (1955). Equating test scores-A maximum likelihood solution. Psychometrika, 20(3), 193–200. https://doi.org/10.1007/BF02289016
25. Lord, F. M. (1980). Applications of Item Response Theory to Practical Testing Problems. Routledge.
26. Lu, R., & Guo, H. (2018). A Simulation Study to Compare Nonequivalent Groups With Anchor Test Equating and Pseudo-Equivalent Group Linking. December. https://doi.org/10.1002/ets2.12196
27. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill
28. OECD. (2019). PISA 2018 technical report. OECD Publishing. https://www.oecd.org/pisa/data/pisa2018technicalreport/ .https://doi.org/10.1787/b25efab8-en
29. OECD. (2020). “Construction of indices”, in PISA 2018 Results (Volume III): What School Life Means for Students’ Lives: Vol. III (Issue Volume III). https://doi.org/10.1787/0a428b07-en
30. Ozsoy, S. N., & Kilmen, S. (2023). Comparison of Kernel equating methods under NEAT and NEC designs. IJATE, 10(1), 56–75. https://doi.org/10.21449/ijate.981367
31. Pokropek, A., & Borgonovi, F. (2020). Linking via Pseudo-Equivalent Group Design: Methodological Considerations and an Application to the PISA and PIAAC Assessments. Journal of Educational Measurement, 57(4), 527–546. https://doi.org/10.1111/jedm.12261
32. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55. https://doi.org/10.1093/biomet/70.1.41
33. Sansivieri, V. (2017). Item response theory equating with the non-equivalent groups with covariates design (Doctoral dissertation, University of Bologna).
34. Seheryeli, M. Yıldırım. (2022). Ortak değişkenlerin özelliklerine göre kernel eşitleme sonuçlarının karşılaştırılması (Doctoral dissertation, Hacettepe University).
35. Sinharay, S., & Holland, P. (2006). The Correlation Between the Scores of a Test and an Anchor Test. ETS Research Report Series, 2006(1), i–28. https://doi.org/10.1002/j.2333-8504.2006.tb02010.x
36. Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Pearson.
37. Tanguma, J. (2000). Equating test scores using the linear method: A primer. Paper Presented at the Annual Meeting of the Southwest Educational Research Association. https://files.eric.ed.gov/fulltext/ED449169.pdf
38. Troyanskaya, O., Cantor, M., Sherlock, G., Brown, P., Hastie, T., Tibshirani, R., Botstein, D., & Altman, R. B. (2001). Missing value estimation methods for DNA microarrays. 17(6), 520–525. https://doi.org/10.1093/bioinformatics/17.6.520
39. Uysal, İ., & Eren, A. (2021). “Eğri bir aynada doğruyu görmek”: PISA 2018 öğrenci başarısının doğrusal olmayan bir çerçevede incelenmesi. Journal of Uludag University Faculty of Education, 34(3), 923–978. https://doi.org/10.19171/uefad.932207
40. von Davier, A. A., Holland, P. W., & Thayer, D. T. (2004a). The chain and post-stratification methods for observed-score equating: Their relationship to population invariance. Journal of Educational Measurement, 41(1), 15–32. https://doi.org/10.1111/j.1745-3984.2004.tb01156.x
41. Von Davier, A. A., Holland, P. W., & Thayer, D. T. (2004b). The kernel method of test equating. Springer Science & Business Media.
42. Wallin, G., & Wiberg, M. (2019). Kernel Equating Using Propensity Scores for Nonequivalent Groups. 44(4), 390–414. https://doi.org/10.3102/1076998619838226
43. Wallin, G., & Wiberg, M. (2023). Model misspecification and robustness of observed-score test equating using propensity scores. Journal of Educational and Behavioral Statistics, 48(5), 603-635. https://doi.org/10.3102/10769986231161575
44. Wiberg, M., & Bränberg, K. (2015). Kernel Equating Under the Non-Equivalent Groups with Covariates Design. Applied Psychological Measurement, 39(5), 349–361. https://doi.org/10.1177/0146621614567939
45. Wright, N. K., & Dorans, N. J. (1993). Using the selection variable for matching or equating 1, 2. ETS Research Report Series, 1993(1), i-22. https://doi.org/10.1002/j.2333-8504.1993.tb01515.x