Automated Essay Scoring Effect on Test Equating Errors in Mixed-format Test

Abstract

Scoring constructed-response items can be highly difficult, time-consuming, and costly in practice. Improvements in computer technology have enabled automated scoring of constructed-response items. However, the application of automated scoring without an investigation of test equating can lead to serious problems. The goal of this study was to score the constructed-response items in mixed-format tests automatically with different test/training data rates and to investigate the indirect effect of these scores on test equating compared with human raters. Bidirectional long-short term memory (BLSTM) was selected as the automated scoring method for the best performance. During the test equating process, methods based on classical test theory and item response theory were utilized. In most of the equating methods, errors of the equating resulting from automated scoring were close to the errors occurring in equating processes conducted by human raters. It was concluded that automated scoring can be applied because it is convenient in terms of equating.

Keywords

• Test equating
• automated scoring
• classical test theory
• item response theory
• mixed format tests

References

1. Adesiji, K. M., Agbonifo, O. C., Adesuyi, A. T., & Olabode, O. (2016). Development of an automated descriptive text-based scoring system. British Journal of Mathematics & Computer Science, 19(4), 1-14. https://doi.org/10.9734/BJMCS/2016/27558
2. Albano, A. D. (2016). equate: An R package for observed-score linking and equating. Journal of Statistical Software, 74(8), 1-36. https://doi.org/10.18637/jss.v074.i08
3. Almond, R. G. (2014). Using automated essay scores as an anchor when equating constructed-response writing tests. International Journal of Testing, 14(1), 73 91. https://doi.org/10.1080/15305058.2013.816309
4. Angoff, W. H. (1984). Scales, norms and equivalent scores. Educational Testing Service.
5. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® v.2. Journal of Technology, Learning, and Assessment, 4(3), 1-30. http://www.jtla.org.
6. Barendse, M. T., Oort, F. J., & Timmerman, M. E. (2015). Using exploratory factor analysis to determine the dimensionality of discrete responses. Structural Equation Modeling: A Multidisciplinary Journal, 22(1), 87 101. https://doi.org/10.1080/10705511.2014.934850
7. Bartlett, M. S. (1950). Tests of significance in factor analysis. British Journal of Psychology, 3(2), 77-85. https://doi.org/10.1111/j.2044-8317.1950.tb00285.x
8. Chen, H., Xu, J., & He, B. (2014). Automated essay scoring by capturing relative writing quality. The Computer Journal, 57(9), 1318-1330. https://doi.org/10.1093/comjnl/bxt117

9. Cliff, N. (1993). Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin, 114(3), 494-509. https://doi.org/10.1037/0033-2909.114.3.494
10. Cliff, N. (1996). Ordinal methods for behavioral data analysis. Routledge.
11. Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. The Journal of Applied Psychology, 78(1), 98 104. https://doi.org/10.1037/0021 9010.78.1.98
12. Creswell, J. W. (2012). Educational research: Planning, conducting and evaluating quantitative and qualitative research (4th ed.). Pearson.
13. Deng, W., & Monfils, R. (2017). Long-term impact of valid case criterion on capturing population-level growth under Item Response Theory equating (Research Report 17-17). Educational Testing Service. https://doi.org/10.1002/ets2.12144
14. Fritz, C. O., Morris, P. E., & Richler, J. J. (2012). Effect size estimates: Current use, calculations, and interpretation. Journal of Experimental Psychology: General, 141(1), 2-18. https://doi.org/10.1037/a0024338
15. Gonzalez, J., & Wiberg, M. (2017). Applying test equating methods: Using R. Springer.
16. Hagge, S. L., & Kolen, M. J. (2011). Equating mixed-format tests with format representative and non-representative common items. In M. J. Kolen & W-C. Lee (Eds.), Mixed-format tests: Psychometric properties with a primary focus on equating (Vol. 1, pp. 95-135). Center for Advanced Studies in Measurement and Assessment, The University of Iowa.
17. Hagge, S. L., Liu, C., He, Y., Powers, S. J., Wang, W., & Kolen, M. J. (2011). A comparison of IRT and traditional equipercentile methods in mixed-format equating. In M. J. Kolen & W-C. Lee (Eds.), Mixed-format tests: Psychometric properties with a primary focus on equating (Vol. 1, pp. 19-50). Center for Advanced Studies in Measurement and Assessment, The University of Iowa.
18. He, Y. (2011). Evaluating equating properties for mixed-format tests [Doctoral dissertation, University of IOWA]. https://ir.uiowa.edu/etd/981/
19. Kaiser, H. F. (1970). A second generation little jiffy. Psychometrika, 35(4), 401 415. https://doi.org/10.1007/BF02291817
20. Kaiser, H. F., & Rice, J. (1974). Little jiffy, mark IV. Educational and Psychological Measurement, 34(1), 111–117. https://doi.org/10.1177/001316447403400115
21. Kolen, M. J., & Brennan, R. L. (2014). Test equating, scaling and linking (2nd ed.). Springer.
22. Keller, L. A., & Keller, R. R. (2011). The long-term sustainability of different Item Response Theory scaling methods. Educational and Psychological Measurement, 71(2), 362–379. https://doi.org/10.1177/0013164410375111
23. LaFlair, G. T., Isbell, D., May, L. D. N., Arvizu, M. N. G., & Jamieson, J. (2017). Equating in small scale language testing programs. Language Testing, 34(1), 127 144. https://doi.org/10.1177/0265532215620825
24. Lee, E., Lee, W-C., & Brennan, R. L. (2012). Exploring equity properties in equating using AP® examinations (Report No. 2012-4). CollegeBoard.
25. Liu, C., & Kolen, M. J. (2011). A comparison among IRT equating methods and traditional equating methods for mixed-format tests. In M. J. Kolen & W-C. Lee (Eds.), Mixed-format tests: Psychometric properties with a primary focus on equating (Vol. 1, pp. 75-94). Center for Advanced Studies in Measurement and Assessment, The University of Iowa.
26. Lorenzo-Seva, U., & Ferrando, P. J. (2006). FACTOR: A computer program to fit the exploratory factor analysis model. Behavior Research Methods, 38(1), 88-91. https://doi.org/10.3758/BF03192753
27. Martinez, M. E. (1999). Cognition and the question of test item format. Educational Psychologist, 34(4), 207-218. https://doi.org/10.1207/s15326985ep3404_2
28. Messick, S. (1993). Trait equivalence as construct validity of score interpretation across multiple methods of measurement. In R. E. Bennett & W. C. Ward (Eds.), Construction versus choice in cognitive measurement: Issues in constructed-response, performance testing, and portfolio assessment (pp. 61-73). Lawrence Erlbaum Associates, Inc.
29. MoNE. (2017). Monitoring and evaluation of academic skills (ABİDE) 2016 8th grade report. https://odsgm.meb.gov.tr/meb_iys_dosyalar/2017_11/30114819_iY-web-v6.pdf
30. Moses, T. P., & von Davier, A. A. (2006). An SAS macro for loglinear smoothing: Applications and implications (Report No. 06-05). Educational Testing Service.
31. Moses, T., von Davier, A. A., & Casabianca, J. (2004). Loglinear smoothing: An alternative numerical approach using SAS (Research No. 04-27). Educational Testing Service.
32. Muthén, L. K., & Muthén, B. O. (2012). Mplus user’s guide (7th ed.). Muthén & Muthén.
33. Olgar, S. (2015). The integration of automated essay scoring systems into the equating process for mixed format tests [Doctoral dissertation, Florida State University]. http://diginole.lib.fsu.edu/islandora/object/fsu%3A253122
34. Page, E. B. (1966). The imminence of grading essays by computers. Phi Delta Kappan, 47(5), 238–243. http://www.jstor.org/stable/20371545
35. Pang, X., Madera, E., Radwan, N., & Zhang, S. (2010). A comparison of four test equating methods (Research Report). Eqao.
36. R Development Core Team. (2018). R: A language and environment for statistical computing (version 3.5.2) [Computer software]. R Foundation for Statistical Computing.
37. Rodriguez, M. C. (2002). Choosing an item format. In G. Tindal & T. M. Haladyna (Eds.), Large scale assessment programs for all students: Validity, technical adequacy and implementation (pp. 213-231). Lawrence Erlbaum Associates.
38. SAS Instutite. (2015). Statistical analysis software (version 9.4) [Computer software]. SAS Institute.
39. Spence, P. D. (1996). The effect of multidimensionality on unidimensional equating with item response theory [Doctoral dissertation, University of Florida]. https://www.proquest.com/docview/304315473
40. Tankersley, K. (2007). Tests that teach: Using standardized tests to improve instruction. Association for Supervision and Curriculum Development.
41. Tate, R. (2000). Performance of a proposed method for the linking of mixed-format tests with constructed-response and multiple-choice items. Journal of Educational Measurement, 37(4), 329-346. http://www.jstor.org/stable/1435244
42. Timmerman, M. E., & Lorenzo-Seva, U. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16(2), 209 220. https://doi.org/10.1037/a0023353
43. Torchiano, M. (2020). effsize: Efficient Effect Size Computation (Version 0.8.1) [Computer software]. https://CRAN.R-project.org/package=effsize
44. Weiss, D. J., & Minden, S. V. (2012). A comparison of item parameter estimates from Xcalibre 4.1 and Bilog-MG (Technical Report). Assessment Systems Corporation.
45. Wolf, R. (2013). Assessing the impact of characteristics of the test, common-items, and examinees on the preservation of equity properties in mixed-format test equating [Doctoral dissertation, University of Pittsburgh]. https://core.ac.uk/download/pdf/19441049.pdf
46. Yoes, M. E. (1996). User's manual for the XCALIBRE marginal maximum-likelihood estimation program [Computer software]. Assessment Systems Corporation.
47. Zu, J., & Liu, J. (2010). Observed score equating using discrete and passage-based anchor items. Journal of Educational Measurement, 47(4), 395 412. https://doi.org/10.1111/j.1745-3984.2010.00120.x