The Comparison of the Equated Tests Scores by Using Various Covariates using Bayesian Nonparametric Model

Year 2021, Volume: 12 Issue: 2, 192 - 211, 30.06.2021

Meltem Yurtçu , Hülya Kelecioğlu , Edward Boone

https://doi.org/10.21031/epod.864744

Cited By: 1

Abstract

This research is based on obtaining equated scores by using covariates in the Bayesian nonparametric model. As covariates in the study, gender, mathematics self-efficacy scores, and common item scores were used. The distributions were obtained for all score groups. Hellinger Distance was calculated to obtain the distances between the distributions of equated scores by using covariates and the distribution of the target test scores. These distances were compared with the distributions of equated scores obtained from methods based on Item Response Theory. The study was conducted on Canadian and Italian samples of Programme for International Student Assessment (PISA) 2012. PARSCALE and IRTEQ were used for classical methods, and R was used for Bayesian nonparametric model. When gender, mathematics self-efficacy scores, and common item scores were used as covariates in the model, distance values of obtained equated scores to target test scores were close to each other, but their distributions were different. The closest distribution to target test scores was achieved when gender and mathematics self-efficacy scores were used together as covariates in the model, and the farthest distributions were obtained from item response theory methods. As a result of the research, it was determined that the model is more informative than the classical methods.

Keywords

Test equating, Bayesian nonparametric model, covariates, equated scores, score distribution

References

• Angoff, W. H. (1971). Scales, norms and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (pp. 508-600). Washington, DC: American Council on Education.
• Ayotola, A., & Adedeji, T. (2009). The relationship between mathematics self-efficacy and achievement in mathematics. Procedia Social and Behavioral Science, 1, 953-957. Retrieved from https://cyberleninka.org/article/n/1232855.pdf
• Barrientos, A. F., Jara, A., & Quintana, F. (2012). On the support of MacEachern’s dependent dirichlet processes and extensions. Bayesian Analaysis, 7(2), 277-310. Retrieved from https://projecteuclid.org/download/pdfview_1/euclid.ba/1339878889
• Barrientos, A. F., Jara, A., & Quintana, F. (2016). Fully nonparametric regression for bounded data using Bernstein polynomials. Retrieved from http://www.mat.uc.cl/~ajara/Publications_files/DependentBernstein.pdf
• Berger, J. O., Boukai, B., & Wang, Y. (1997). Unied frequentist and bayesian testing of a precise hypothesis. Statistical Science, 12(3), 133-160. Retrieved from https://www2.stat.duke.edu/~berger/papers/statsci.pdf
• Boone, E. L. Merrick, J. R. W., & Krachey, M. J. (2012). A Hellinger distance approach to MCMC diagnostics. Journal of Statistical Computation and Simulation, 84(4), 833-849. doi: 10.1080/00949655.2012.729588
• Branberg, K., & Wiberg, M. (2011). Observed score linear equating with covariates. Journal of Educational Measurement, 48(4), 419-440. doi: 10.1111/j.1745-3984.2011.00153.x
• De Iorio, M., Müller, P., Rosner, G., L., & MacEachern, S. N. (2004). An ANOVA model for dependent random measures. Journal of the American Statistical Association, 99(465), 205-215. doi: 10.1198/016214504000000205
• Ding, Y. (2016). How do students’ mathematics self-efficacy, mathematics self-concept and mathematics anxiety influence mathematical literacy?-A comparison between Shanghai-China and Sweden in PISA 2012 (Master thesis). University of Gothenburg, Faculty of Education, Gothenburg, Sweden.
• Dorans, J. N., & Holland, P. W. (2000). Population invariance and the equitability of tests: Basic theory and the linear case. Journal of Educational Measurement, 37(4), 281-306. doi: 10.1111/j.1745-3984.2000.tb01088.x
• Dorans, N. J., Moses, T. P., & Eignor, D. R. (2010). Principles and practices of test score equating (ETS RR-10-29). New Jersey: ETS, Princeton.
• González J., & Wiberg M. (2017) Recent developments in equating. In J. González & M. Wiberg (Eds.), Applying test equating methods: Methodology of educational measurement and assessment (pp. 157-178). Switzerland: Springer, Cham
• Gonzalez, J., Barrientos, A. F., & Quintana, F. A. (2015a). Bayesian nonparametric estimation of test equating functions with covariates. Computational Statistics and Data Analysis 89, 222-244. doi: 10.1016/j.csda.2015.03.012
• Gonzalez, J., Barrientos, A. F., & Quintana, F. A. (2015b). A dependent Bayesian nonparametric model for test equating. In R. E. Millsap, D. M. Bolt, L. A. van der Ark, & W-C. Wang, (Eds.) Quantitative psychology research (pp. 213-226). New York: Springer Cham Heidelberg New York Dordrecht London.
• Hackett, G., & Betz, N. E. (1989). An exploration of the mathematics self-efficacy/mathematics performance correspondence. Journal for Research in Mathematics Education, 20(3), 261-273. doi: 10.2307/749515
• Hall, C. W., & Hoff, C. (1988). Gender differences in mathematical performance. Educational Studies in Mathematics 19(1988) 395-401. Retrieved from https://link.springer.com/content/pdf/10.1007%2FBF00312455.pdf
• Karabatsos, G., & Walker, S. G. (2009). A bayesian nonparametric approach to test equating. Psychometrika, 74(2), 211-232. doi: 10.1007/S11336-008-9096-6
• Kelley, T. L. (1939). The selection of upper and lower groups for the validation of test items. Journal of Educational Psychology, 30, 17-24.
• Kim, S., Livingston, S. A., & Lewis, C. (2009). Effectiveness of collateral information for improving equating in small samples. New Jersey: ETS, Princeton.
• Kim, S., Livingston, S. A., & Lewis, C. (2011). Collateral information for equating in small samples: A preliminary investigation. Applied Measurement in Education, 24(4), 302-323. doi: 10.1080/08957347.2011.607057
• Koğar, H. (2015). PISA 2012 matematik okuryazarlığını etkileyen faktörlerin aracılık modeli ile incelenmesi. Eğitim ve Bilim, 40(179), 45-55. doi: 10.15390/EB.2015.4445
• Kolen, M. J. (1988). Traditional equating methodology. Educational Measurement: Issues and Practice, 7(4), 29-36. doi: 10.1111/j.1745-3992.1988.tb00843.x
• Kolen, M. J., & Brennan, R. L. (2014). Test equating, scaling, and linking: Methods and practices (3nd. ed.). New York: Springer.
• Kruschke, J. K. (2010). Bayesian data analysis. Wiley Interdisciplinary Reviews; Cognitive Science, 1(5), 658-676, doi: 10.1002/wcs.72
• Kruschke, J. K. (2015). Doing Bayesian data analysis (Second Ed.): A tutorial with R, JAGS, and Stan. Waltham, MA: Academic Press / Elsevier.
• Kruschke, J. K., Aguinis, H., & Joo, H. (2012). The Time has come: Bayesian methods for data analysis in the organizational sciences. Organizational Research Methods, 15(4) 722-752. doi: 10.1177/1094428112457829
• Lee, A. H., & Boone, E. L. (2011). A frequentist assessment of Bayesian inclusion probabilities for screening predictors. Journal of Statistical Computation and Simulation, 81(9), 1111-1119. doi: 10.1080/00949651003702135
• Li, D., Jiang, Y., & von Davier, A. A. (2012). The accuracy and consistency of a series of IRT true score equatings. Journal of Educational Mesurment, 49(2), 167-189. doi: 10.1111/j.1745-3984.2012.00167.x
• Lindberg, S. M., Hyde, J. S., Petersen, J. L., & Linn, M. C. (2010). New trends in gender and mathematics performance: A meta-analysis. Psychological Bulletin, 136(6), 1123-1135. doi: 10.1037/a0021276
• Liou, M. (1998). Establishing score comparability in heterogeneous populations. Statistica Sinica, 8, 669-690. Retrieved from http://www3.stat.sinica.edu.tw/statistica/oldpdf/A8n33.pdf
• Liou, M., Cheng, P. E., & Li, M. (2001). Estimating comparable scores using surrogate variables. Applied Psychological Measurement, 25(2), 197-207. doi: 10.1177/01466210122032000
• Livingston, S. A. (2004). Equating test scores (Without IRT). Educational Testing Service. Retrieved from https://www.ets.org/Media/Research/pdf/LIVINGSTON.pdf
• Livingston, S. A., & Lewis, C. (2009). Small-sample equating with prior information. (ETS Research Rep. No. RR-09-25). New Jersey: ETS, Princeton.
• MacEachern, S. N. (1999). Dependent nonparametric processes. Retrieved from https://people.eecs.berkeley.edu/~russell/classes/cs294/f05/papers/maceachern-1999.pdf
• MacEachern, S.N., (2000). Dependent Dirichlet processes (Tech. rep). Department of Statistics, The Ohio State University. Retrieved from https://people.eecs.berkeley.edu/~russell/classes/cs294/f05/papers/maceachern-1999.pdf
• Martin, M. O., Mullis, I. V. S., Foy, P., & Stanco, G. M. (2012). TIMSS 2011 international results in science. Boston College, MA, USA: International Study Center.
• Mittelhaeuser, M.-A., Beguin, A. A., & Sijtsma, K. (2011). Comparing the effectiveness of different linking design: The internal anchor versus the external anchor and pre-test data (Measurement and Research Department Reports, 1). Arnhem: Cito.
• Moses, T., Deng, W., & Zhang, Y.-L. (2010). The use of two anchors in nonequivalent groups with anchor test (NEAT) equating (RR-10-23). New Jersey: ETS, Princeton.
• Müller, P., & Quintana, F. A. (2004). Nonparametric Bayesian data analysis. Statistical Science, 19(1), 95-110. doi: 10.1214/088342304000000017
• Oh, H. J., Guo, H., & Walker, M. E. (2009). Impraved reability estimates for small samples using empirical Bayes teshniques (RR-09-46). New Jersey: ETS, Princeton.
• Orbanz, P., & Teh, Y. W.(2010). Bayesian nonparametric models. In C. Sammut & G. I. Webb (Eds.), Encyclopedia of Machine Learning. Boston, MA: Springer. doi: 10.1007/978-0-387-30164-8_66
• Petrone, S. (1999a). Bayesian density estimation using Bernstein polynomials. The Canadian Journal of Statistics 27(Varsa sayı no) 105-126. Retrieved from https://www.jstor.org/stable/pdf/3315494.pdf?refreqid=excelsior%3A7e6e0614f5a5f181dfd25d2ad6947bc6
• Petrone, S. (1999b). Random Bernstein polynomials. Scandinavian Journal of Statistics 26, 373-393. Retrieved from https://www.jstor.org/stable/pdf/4616563.pdf?refreqid=excelsior%3A801798d1ac07988dafb6e83769c949b2
• Rounder, J. N., Morey, R. D., Speckman, P. L., & Province, M. (2012). Default Bayes factors for ANOVA designs. Journal of Mathematical Psychology, 56(2012), 356-374, doi: 10.1016/j.jmp.2012.08.001
• Schulz, W. (2005, April). Mathematics self-efficacy and student expectations: Result from PISA 2003. Annual Meetings of the American Educational Research Association in Montreal. Retrieved from https://files.eric.ed.gov/fulltext/ED490044.pdf
• Shah, A., & Ghahramani, Z. (2013, September). Determinantal clustering process- A nonparametric bayesian approach to kernel based semi-supervised clustering. Proceedings of the TwentyNinth Conference on Uncertainty in Artificial Intelligence. Retrieved from http://auai.org/uai2013/prints/papers/200.pdf
• Siegle, D., & McCoach, D. B. (2007). Increasing student mathematics self-efficacy through teacher training. Journal of Advanced Academics, 18(2), 278-312. Retrieved from https://files.eric.ed.gov/fulltext/EJ767452.pdf
• Sinharay, S., & Holland, P. W. (2006). Choice of anchor test in equating (RR-06-35). New Jersey: ETS, Princeton.
• StataCorp. (2015). Stata Bayesian analysis reference manual release 14. College Station, TX: StataCorp LLC. https://www.stata.com/manuals14/bayes.pdf
• Thien, L. R., & Darmawan, I. G. N. (2016). Factors associated with Malaysian mathematics Performance in PISA 2012. In L. M. Thien, N. A. Razak, J. Keeves, & I. G. N. Darmawan (Eds.), What can PISA 2012 data tell us?: Performance and challenges in five participating Southeast Asian countries (pp. 81-105). Rotterdam: Sense Publisher.
• van de Schoot, R., Kaplan, D., Denissen, J., Asendorpf, J. B., Neyer, F. J., & van Aken, M. A. G. (2013). A gentle introduction to Bayesian analysis: Applications to developmental research. Child Development, 85(3), 1-19. doi: 10.1111/cdev.12169
• Wallin, G., & Wiberg, M. (2017). Non-equivalent groups with covariates design using propensity scores for kernel equating. In L. A. van der Ark, M. Wiberg, S. A. Culpepper, J. A. Douglas, & W.-C. Wang (Eds.), Quantitative psychology – 81st annual meeting of the psychometric society, Asheville, North Carolina. New York: Springer.
• Wei, H. (2010, May). Impact of non-representative anchor items on scale stability. Paper presented at the Annual Conference of the National Council on Measurement in Education, Denver, CO.
• Wiberg, M. (2015). Anote on equating test scores with covariates. In E. Frackle-Fornius (Ed.), Festschrift in honor of Hans Nyquist on the occasion of his 65th birthday (pp. 96-99). Stockholm: Department of Statistics Stockholm University, Sweden.
• Wiberg, M., & Gonzalez, J. (2016). Statistical assessment of estimated transformations in observed-score equating. Journal of Educational Measurement. 53(1), 106-125. Retrieved from: http://www.mat.uc.cl/~jorge.gonzalez/papers/TR/Assess_TR.pdf
• Wiberg, M., & von Davier, A. A. (2017). Examining the impact of covariates on anchor tests to ascertain quality over time in a college admissions test. International Journal of Testing, 17(2), 105-126. doi: 10.1080/15305058.2016.1277357
• Wiberg, M., & Branberg, K. (2015). Kernel equating under the non-equivalent groups with covariates design. Applied Psychological Measurement, 39(5), 349-361. doi: 10.1177/0146621614567939
• Wright, N. K., & Dorans, N. J. (1993). Using the selection variable for matching or equating (RR-93–04). New Jersey: ETS, Princeton.
• Yıldırım, H. H., Yıldırım, S., Yetişir , M. İ., & Ceylan, E. (2013). PISA 2012 ulusal ön raporu. Ankara: MEB Yenilik ve Eğitim Teknolojileri Genel Müdürlüğü (YeğiTek).