The Comparison of the Equated Tests Scores by Using Various Covariates using Bayesian Nonparametric Model

Abstract

This research is based on obtaining equated scores by using covariates in the Bayesian nonparametric model. As covariates in the study, gender, mathematics self-efficacy scores, and common item scores were used. The distributions were obtained for all score groups. Hellinger Distance was calculated to obtain the distances between the distributions of equated scores by using covariates and the distribution of the target test scores. These distances were compared with the distributions of equated scores obtained from methods based on Item Response Theory. The study was conducted on Canadian and Italian samples of Programme for International Student Assessment (PISA) 2012. PARSCALE and IRTEQ were used for classical methods, and R was used for Bayesian nonparametric model. When gender, mathematics self-efficacy scores, and common item scores were used as covariates in the model, distance values of obtained equated scores to target test scores were close to each other, but their distributions were different. The closest distribution to target test scores was achieved when gender and mathematics self-efficacy scores were used together as covariates in the model, and the farthest distributions were obtained from item response theory methods. As a result of the research, it was determined that the model is more informative than the classical methods.

Keywords

• Test equating
• Bayesian nonparametric model
• covariates
• equated scores
• score distribution

References

1. Angoff, W. H. (1971). Scales, norms and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (pp. 508-600). Washington, DC: American Council on Education.
2. Ayotola, A., & Adedeji, T. (2009). The relationship between mathematics self-efficacy and achievement in mathematics. Procedia Social and Behavioral Science, 1, 953-957. Retrieved from https://cyberleninka.org/article/n/1232855.pdf
3. Barrientos, A. F., Jara, A., & Quintana, F. (2012). On the support of MacEachern’s dependent dirichlet processes and extensions. Bayesian Analaysis, 7(2), 277-310. Retrieved from https://projecteuclid.org/download/pdfview_1/euclid.ba/1339878889
4. Barrientos, A. F., Jara, A., & Quintana, F. (2016). Fully nonparametric regression for bounded data using Bernstein polynomials. Retrieved from http://www.mat.uc.cl/~ajara/Publications_files/DependentBernstein.pdf
5. Berger, J. O., Boukai, B., & Wang, Y. (1997). Unied frequentist and bayesian testing of a precise hypothesis. Statistical Science, 12(3), 133-160. Retrieved from https://www2.stat.duke.edu/~berger/papers/statsci.pdf
6. Boone, E. L. Merrick, J. R. W., & Krachey, M. J. (2012). A Hellinger distance approach to MCMC diagnostics. Journal of Statistical Computation and Simulation, 84(4), 833-849. doi: 10.1080/00949655.2012.729588
7. Branberg, K., & Wiberg, M. (2011). Observed score linear equating with covariates. Journal of Educational Measurement, 48(4), 419-440. doi: 10.1111/j.1745-3984.2011.00153.x
8. De Iorio, M., Müller, P., Rosner, G., L., & MacEachern, S. N. (2004). An ANOVA model for dependent random measures. Journal of the American Statistical Association, 99(465), 205-215. doi: 10.1198/016214504000000205

9. Ding, Y. (2016). How do students’ mathematics self-efficacy, mathematics self-concept and mathematics anxiety influence mathematical literacy?-A comparison between Shanghai-China and Sweden in PISA 2012 (Master thesis). University of Gothenburg, Faculty of Education, Gothenburg, Sweden.
10. Dorans, J. N., & Holland, P. W. (2000). Population invariance and the equitability of tests: Basic theory and the linear case. Journal of Educational Measurement, 37(4), 281-306. doi: 10.1111/j.1745-3984.2000.tb01088.x
11. Dorans, N. J., Moses, T. P., & Eignor, D. R. (2010). Principles and practices of test score equating (ETS RR-10-29). New Jersey: ETS, Princeton.
12. González J., & Wiberg M. (2017) Recent developments in equating. In J. González & M. Wiberg (Eds.), Applying test equating methods: Methodology of educational measurement and assessment (pp. 157-178). Switzerland: Springer, Cham
13. Gonzalez, J., Barrientos, A. F., & Quintana, F. A. (2015a). Bayesian nonparametric estimation of test equating functions with covariates. Computational Statistics and Data Analysis 89, 222-244. doi: 10.1016/j.csda.2015.03.012
14. Gonzalez, J., Barrientos, A. F., & Quintana, F. A. (2015b). A dependent Bayesian nonparametric model for test equating. In R. E. Millsap, D. M. Bolt, L. A. van der Ark, & W-C. Wang, (Eds.) Quantitative psychology research (pp. 213-226). New York: Springer Cham Heidelberg New York Dordrecht London.
15. Hackett, G., & Betz, N. E. (1989). An exploration of the mathematics self-efficacy/mathematics performance correspondence. Journal for Research in Mathematics Education, 20(3), 261-273. doi: 10.2307/749515
16. Hall, C. W., & Hoff, C. (1988). Gender differences in mathematical performance. Educational Studies in Mathematics 19(1988) 395-401. Retrieved from https://link.springer.com/content/pdf/10.1007%2FBF00312455.pdf
17. Karabatsos, G., & Walker, S. G. (2009). A bayesian nonparametric approach to test equating. Psychometrika, 74(2), 211-232. doi: 10.1007/S11336-008-9096-6
18. Kelley, T. L. (1939). The selection of upper and lower groups for the validation of test items. Journal of Educational Psychology, 30, 17-24.
19. Kim, S., Livingston, S. A., & Lewis, C. (2009). Effectiveness of collateral information for improving equating in small samples. New Jersey: ETS, Princeton.
20. Kim, S., Livingston, S. A., & Lewis, C. (2011). Collateral information for equating in small samples: A preliminary investigation. Applied Measurement in Education, 24(4), 302-323. doi: 10.1080/08957347.2011.607057
21. Koğar, H. (2015). PISA 2012 matematik okuryazarlığını etkileyen faktörlerin aracılık modeli ile incelenmesi. Eğitim ve Bilim, 40(179), 45-55. doi: 10.15390/EB.2015.4445
22. Kolen, M. J. (1988). Traditional equating methodology. Educational Measurement: Issues and Practice, 7(4), 29-36. doi: 10.1111/j.1745-3992.1988.tb00843.x
23. Kolen, M. J., & Brennan, R. L. (2014). Test equating, scaling, and linking: Methods and practices (3nd. ed.). New York: Springer.
24. Kruschke, J. K. (2010). Bayesian data analysis. Wiley Interdisciplinary Reviews; Cognitive Science, 1(5), 658-676, doi: 10.1002/wcs.72
25. Kruschke, J. K. (2015). Doing Bayesian data analysis (Second Ed.): A tutorial with R, JAGS, and Stan. Waltham, MA: Academic Press / Elsevier.
26. Kruschke, J. K., Aguinis, H., & Joo, H. (2012). The Time has come: Bayesian methods for data analysis in the organizational sciences. Organizational Research Methods, 15(4) 722-752. doi: 10.1177/1094428112457829
27. Lee, A. H., & Boone, E. L. (2011). A frequentist assessment of Bayesian inclusion probabilities for screening predictors. Journal of Statistical Computation and Simulation, 81(9), 1111-1119. doi: 10.1080/00949651003702135
28. Li, D., Jiang, Y., & von Davier, A. A. (2012). The accuracy and consistency of a series of IRT true score equatings. Journal of Educational Mesurment, 49(2), 167-189. doi: 10.1111/j.1745-3984.2012.00167.x
29. Lindberg, S. M., Hyde, J. S., Petersen, J. L., & Linn, M. C. (2010). New trends in gender and mathematics performance: A meta-analysis. Psychological Bulletin, 136(6), 1123-1135. doi: 10.1037/a0021276
30. Liou, M. (1998). Establishing score comparability in heterogeneous populations. Statistica Sinica, 8, 669-690. Retrieved from http://www3.stat.sinica.edu.tw/statistica/oldpdf/A8n33.pdf
31. Liou, M., Cheng, P. E., & Li, M. (2001). Estimating comparable scores using surrogate variables. Applied Psychological Measurement, 25(2), 197-207. doi: 10.1177/01466210122032000
32. Livingston, S. A. (2004). Equating test scores (Without IRT). Educational Testing Service. Retrieved from https://www.ets.org/Media/Research/pdf/LIVINGSTON.pdf
33. Livingston, S. A., & Lewis, C. (2009). Small-sample equating with prior information. (ETS Research Rep. No. RR-09-25). New Jersey: ETS, Princeton.
34. MacEachern, S. N. (1999). Dependent nonparametric processes. Retrieved from https://people.eecs.berkeley.edu/~russell/classes/cs294/f05/papers/maceachern-1999.pdf
35. MacEachern, S.N., (2000). Dependent Dirichlet processes (Tech. rep). Department of Statistics, The Ohio State University. Retrieved from https://people.eecs.berkeley.edu/~russell/classes/cs294/f05/papers/maceachern-1999.pdf
36. Martin, M. O., Mullis, I. V. S., Foy, P., & Stanco, G. M. (2012). TIMSS 2011 international results in science. Boston College, MA, USA: International Study Center.
37. Mittelhaeuser, M.-A., Beguin, A. A., & Sijtsma, K. (2011). Comparing the effectiveness of different linking design: The internal anchor versus the external anchor and pre-test data (Measurement and Research Department Reports, 1). Arnhem: Cito.
38. Moses, T., Deng, W., & Zhang, Y.-L. (2010). The use of two anchors in nonequivalent groups with anchor test (NEAT) equating (RR-10-23). New Jersey: ETS, Princeton.
39. Müller, P., & Quintana, F. A. (2004). Nonparametric Bayesian data analysis. Statistical Science, 19(1), 95-110. doi: 10.1214/088342304000000017
40. Oh, H. J., Guo, H., & Walker, M. E. (2009). Impraved reability estimates for small samples using empirical Bayes teshniques (RR-09-46). New Jersey: ETS, Princeton.
41. Orbanz, P., & Teh, Y. W.(2010). Bayesian nonparametric models. In C. Sammut & G. I. Webb (Eds.), Encyclopedia of Machine Learning. Boston, MA: Springer. doi: 10.1007/978-0-387-30164-8_66
42. Petrone, S. (1999a). Bayesian density estimation using Bernstein polynomials. The Canadian Journal of Statistics 27(Varsa sayı no) 105-126. Retrieved from https://www.jstor.org/stable/pdf/3315494.pdf?refreqid=excelsior%3A7e6e0614f5a5f181dfd25d2ad6947bc6
43. Petrone, S. (1999b). Random Bernstein polynomials. Scandinavian Journal of Statistics 26, 373-393. Retrieved from https://www.jstor.org/stable/pdf/4616563.pdf?refreqid=excelsior%3A801798d1ac07988dafb6e83769c949b2
44. Rounder, J. N., Morey, R. D., Speckman, P. L., & Province, M. (2012). Default Bayes factors for ANOVA designs. Journal of Mathematical Psychology, 56(2012), 356-374, doi: 10.1016/j.jmp.2012.08.001
45. Schulz, W. (2005, April). Mathematics self-efficacy and student expectations: Result from PISA 2003. Annual Meetings of the American Educational Research Association in Montreal. Retrieved from https://files.eric.ed.gov/fulltext/ED490044.pdf
46. Shah, A., & Ghahramani, Z. (2013, September). Determinantal clustering process- A nonparametric bayesian approach to kernel based semi-supervised clustering. Proceedings of the TwentyNinth Conference on Uncertainty in Artificial Intelligence. Retrieved from http://auai.org/uai2013/prints/papers/200.pdf
47. Siegle, D., & McCoach, D. B. (2007). Increasing student mathematics self-efficacy through teacher training. Journal of Advanced Academics, 18(2), 278-312. Retrieved from https://files.eric.ed.gov/fulltext/EJ767452.pdf
48. Sinharay, S., & Holland, P. W. (2006). Choice of anchor test in equating (RR-06-35). New Jersey: ETS, Princeton.
49. StataCorp. (2015). Stata Bayesian analysis reference manual release 14. College Station, TX: StataCorp LLC. https://www.stata.com/manuals14/bayes.pdf
50. Thien, L. R., & Darmawan, I. G. N. (2016). Factors associated with Malaysian mathematics Performance in PISA 2012. In L. M. Thien, N. A. Razak, J. Keeves, & I. G. N. Darmawan (Eds.), What can PISA 2012 data tell us?: Performance and challenges in five participating Southeast Asian countries (pp. 81-105). Rotterdam: Sense Publisher.
51. van de Schoot, R., Kaplan, D., Denissen, J., Asendorpf, J. B., Neyer, F. J., & van Aken, M. A. G. (2013). A gentle introduction to Bayesian analysis: Applications to developmental research. Child Development, 85(3), 1-19. doi: 10.1111/cdev.12169
52. Wallin, G., & Wiberg, M. (2017). Non-equivalent groups with covariates design using propensity scores for kernel equating. In L. A. van der Ark, M. Wiberg, S. A. Culpepper, J. A. Douglas, & W.-C. Wang (Eds.), Quantitative psychology – 81st annual meeting of the psychometric society, Asheville, North Carolina. New York: Springer.
53. Wei, H. (2010, May). Impact of non-representative anchor items on scale stability. Paper presented at the Annual Conference of the National Council on Measurement in Education, Denver, CO.
54. Wiberg, M. (2015). Anote on equating test scores with covariates. In E. Frackle-Fornius (Ed.), Festschrift in honor of Hans Nyquist on the occasion of his 65th birthday (pp. 96-99). Stockholm: Department of Statistics Stockholm University, Sweden.
55. Wiberg, M., & Gonzalez, J. (2016). Statistical assessment of estimated transformations in observed-score equating. Journal of Educational Measurement. 53(1), 106-125. Retrieved from: http://www.mat.uc.cl/~jorge.gonzalez/papers/TR/Assess_TR.pdf
56. Wiberg, M., & von Davier, A. A. (2017). Examining the impact of covariates on anchor tests to ascertain quality over time in a college admissions test. International Journal of Testing, 17(2), 105-126. doi: 10.1080/15305058.2016.1277357
57. Wiberg, M., & Branberg, K. (2015). Kernel equating under the non-equivalent groups with covariates design. Applied Psychological Measurement, 39(5), 349-361. doi: 10.1177/0146621614567939
58. Wright, N. K., & Dorans, N. J. (1993). Using the selection variable for matching or equating (RR-93–04). New Jersey: ETS, Princeton.
59. Yıldırım, H. H., Yıldırım, S., Yetişir , M. İ., & Ceylan, E. (2013). PISA 2012 ulusal ön raporu. Ankara: MEB Yenilik ve Eğitim Teknolojileri Genel Müdürlüğü (YeğiTek).