AN AUTOMATED SCORING APPROACH FOR ESSAY QUESTIONS

Year 2014, Volume: 1 , 232 - 236, 01.09.2014

Ahmed Alzahrani Abdulkareem Alzahrani Fawaz Alarfaj Khalid Almohammadi Malek Alrashidi

Abstract

The
automated scoring or evaluation for written student responses have been, and
are still a highly interesting topic for both education and natural language
processing, NLP, researchers alike. With the obvious motivation of the
difficulties teachers face when marking or correcting open essay questions; the
development of automatic scoring methods have recently received much attention.
In this paper, we developed and compared number of NLP techniques that
accomplish this task. The baseline for this study is based on a vector space
model, VSM. Where after normalisation, the baseline-system represents each
essay by a vector, and subsequently calculates its score using the cosine
similarity between it and the vector of the model answer. This baseline is then
compared with the improved model, which takes the document structure into
account. To evaluate our system, we used real essays that submitted for
computer science course. Each essay was independently scored by two teachers,
which we used as our gold standard. The systems' scoring was then compared to
both teachers. A high emphasis was added to the evaluation when the two human
assessors are in agreement. The systems' results show a high and promising
performance.

Keywords

Automated essay scoring, project essay grade, e-pedagogy and e-assessment

References

• Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® V. 2. The Journal of Technology, Learning and Assessment, 4(3). Burstein, J. (2003). The E-rater® scoring engine: Automated essay scoring with natural language processing. Burstein, J., Kukich, K., Wolff, S., Lu, C., Chodorow, M., Braden-Harder, L., & Harris, M. D. (1998). Automated scoring using a hybrid feature identification technique. In Proceedings of the 17th international conference on Computational linguistics-Volume 1 (pp. 206–210). Association for Computational Linguistics. Chung, G. K., & O’Neil, H. F. (1997). Methodological approaches to online scoring of essays. Citeseer. Elliot, S. (2003). IntelliMetric: From here to validity. Automated Essay Scoring: A Cross-Disciplinary Perspective, 71–86. Hamp-Lyons, L. (2001). Fourth Generation Writing. On Second Language Writing, 117. Higgins, D., Burstein, J., & Attali, Y. (2006). Identifying off-topic student essays without topic-specific training data. Natural Language Engineering, 12(2), 145–159. Kukich, K. (2000). Beyond automated essay scoring. IEEE Intelligent Systems, 15(5), 22–27. Landauer, T. K. (2003). Automatic essay assessment, 10(3), 295–308. Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25(2-3), 259–284. Landauer, T. K., Laham, D., & Foltz, P. W. (2003). Automated scoring and annotation of essays with the Intelligent Essay Assessor. Automated Essay Scoring: A Cross-Disciplinary Perspective, 87–112. Learning, V. (2003). How does IntelliMetric score essay responses. RB-929). Newtown, PA: Author. Nitko, A. J. (1996). Educational assessment of students. ERIC. Page, E. B. (1966). The imminence of... grading essays by computer. Phi Delta Kappan, 238–243. Page, E. B. (2003). Project essay grade: PEG. Automated Essay Scoring: A Cross-Disciplinary Perspective, 43–54. Rudner, L. M., & Gagne, P. (2001). An overview of three approaches to scoring written essays by computer. ERIC Clearinghouse on Assessment and Evaluation. Rudner, L. M., Garcia, V., & Welch, C. (2006). An evaluation of IntelliMetricTM essay scoring system. The Journal of Technology, Learning and Assessment, 4(4). Rudner, L. M., & Liang, T. (2002). Automated essay scoring using Bayes’ theorem. The Journal of Technology, Learning and Assessment, 1(2). Shermis, M. D., & Barrera, F. D. (2002). Exit Assessments: Evaluating Writing Ability through Automated Essay Scoring. Shermis, M. D., & Burstein, J. C. (2003). Automated essay scoring: A cross-disciplinary perspective. Routledge. Shermis, M. D., Raymat, M. V., & Barrera, F. (2003). Assessing Writing through the Curriculum with Automated Essay Scoring. Sireci, S. G., & Rizavi, S. (2000). Comparing Computerized and Human Scoring of Students’ Essays. Williamson, D. M. (2009). A framework for implementing automated scoring. In Annual Meeting of the American Educational Research Association and the National Council on Measurement in Education, San Diego, CA. Xie, S., Evanini, K., & Zechner, K. (2012). Exploring content features for automated speech scoring. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 103–111). Association for Computational Linguistics. Zechner, K., & Xi, X. (2008). Towards automatic scoring of a test of spoken language with heterogeneous task types. In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications (pp. 98–106). Association for Computational Linguistics.