Home-Grown Automated Essay Scoring in the Literature Classroom: A Solution for Managing the Crowd?

Abstract

Managing crowded classes in terms of classroom assessment is a difficult task due to the amount of time which needs to be devoted to providing feedback to student products. In this respect, the present study aimed to develop an automated essay scoring environment as a potential means to overcome this problem. Secondarily, the study aimed to test if automatically-given scores would correlate with the scores given by a human rater. A quantitative research design employing a machine learning approach was preferred to meet the aims of the study. The data set to be used for machine learning consisted of 160 scored literary analysis essays written in an English Literature course, each essay analyzing a theme in a given literary work. To train the automated scoring model, LightSide software was used. First, textual features were extracted and filtered. Then, Logistic Regression, SMO, SVO, Logistic Tree and Naïve Bayes text classification algorithms were tested by using 10-Fold Cross-Validation to reach the most accurate model. To see if the scores given by the computer correlated with the scores given by the human rater, Spearman’s Rank Order Correlation Coefficient was calculated. The results showed that none of the algorithms were sufficiently accurate in terms of the scores of the essays within the data set. It was also seen that the scores given by the computer were not significantly correlated with the scores given by the human rater. The findings implied that the size of the data collected in an authentic classroom environment was too small for classification algorithms in terms of automated essay scoring for classroom assessment. 

Keywords

• Automated essay scoring,Literary analysis essay,Classification algorithms,Machine learning

References

1. Abu-Mustafa, Y. S., Magdon-Ismail, M., & Lin, H.-T. (2012). Learning from data (1st ed.). Seattle: AML Book.
2. Anderson, S.E., & Ben Jafaar, S. (2006). Policy trends in Ontario education: 1990-2003. (ICEC Working Paper #1). University of Toronto, Ontario Institute. Retrieved on 22 January 2018 from http://fcis.oise.utoronto.ca/~icec/policytrends.pdf
3. Attali, Y. & Burstein, J. (2006). Automated essay scoring with e-rater V.2. Journal of Technology, Learning, and Assessment (JTLA), 4(3). Retrieved on 11 March 2016 from http://ejournals.bc.edu/ojs/ index.php/jtla/article/view/1650
4. Attali, Y. (2013). Validity and reliability of automated essay scoring. In M.D. Shermis & J.C. Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new directions (pp. 181-198). New York, NY: Routledge.
5. Baker, N. L. (2014). “Get it off my stack”: Teachers’ tools for grading papers. Assessing Writing, 19, 36-50. doi: 10.1016/j.asw.2013.11.005.
6. Barker, T. (2011). An automated individual feedback and marking system: An empirical study. The Electronic Journal of E-Learning, 9(1), 1-14.
7. Bauer, J. (2016). A new approach: Closing the writing gap by using reliable assessment to guide and evaluate cross-curricular argumentative writing (Unpublished master’s thesis). The University of Wisconsin, USA.
8. Brookhart, S.M. & Bronowicz, D.L. (2003). “I don’t like writing. It makes my fingers hurt”: students talk about their classroom assessments. Assessment in Education, 10(2), 221-242.

9. Chih-Min, S. & Li-Yi, W. (2013). Factors affecting English language teachers’ classroom assessment practices: A case study of Singapore secondary schools. NIE research brief series. Retrieved on 12 March 2016 from http://hdl.handle.net/10497/15003
10. Darling-Hammond, L., Chung, R., & Frelow, F. (2002). Variation in teacher preparation: How well do different pathways prepare teachers to teach? Journal of Teacher Education, 53(4), 286-302.
11. Deane, P. (2013). On the relation between automated essay scoring and modern views of the writing construct. Assessing Writing, 18(1), 7-24.
12. Dikli, S. (2006). An overview of automated scoring of essays. The Journal of Technology, Learning and Assessment, 5(1), 1-35.
13. Dornyei, Z. (2007). Research methods in applied linguistics. New York: Oxford University Press.
14. Duncan, C. R. & Noonan, B. (2007). Factors affecting teachers' grading and assessment practices. Alberta Journal of Educational Research, 53(1), 1-21.
15. Fulcher, G. & Davidson, F. (2007). Language testing and assessment: An advanced resource book. London: Routledge.
16. Gavriel, J. (2013). Assessment for learning: A wider (classroom-researched) perspective is important for formative assessment and self-directed learning in general practice. Education for Primary Care, 24(2), 93-96.
17. Greenstein, L. (2010). What teachers really need to know about formative assessment? Alexandria, VA: ASCD
18. Heilman, M. & Madnani, N. (2015). The impact of training data on automated short answer scoring performance. Proceedings from NAACL HLT: The tenth workshop on innovative use of NLP for building educational applications (pp. 81-85). The Association for Computational Linguistics: USA. Retrieved on 13 May 2016 from http://www.cs. rochester.edu/u/tetreaul/bea10proceedings.pdf# page=270
19. Hyland, K. & Hyland, F. (Eds.). (2006). Feedback in second language writing: Contexts and issues. New York: Cambridge University Press. http://dx.doi.org/10.1017/ CBO9781139524742
20. Imaki, J. & Ishihara, S. (2013). Experimenting with a Japanese automated essay scoring system in the L2 Japanese environment. Papers in Language Testing and Assessment, 2(2), 28-47.
21. Jung, E. (2017). A comparison of data mining methods in analyzing educational data. In J. Park, Y. Pan, G. Yi, & V. Loia (Eds.), Advances in computer science and ubiquitous computing CSA-CUTE2016 (pp. 173-178). Singapore: Springer.
22. Kumar, C. S., & Rama Sree, R. J. (2014). An attempt to improve classification accuracy through implementation of bootstrap aggregation with sequential minimal optimization during automated evaluation of descriptive answers. Indian Journal of Science and Technology, 7(9), 1369-1375.
23. Lai, Y.H. (2010). Which do students prefer to evaluate their essays: Peers or computer program? British Journal of Educational Technology, 41(3), 432-454.
24. Landauer, T. K., Laham, D. & Foltz, P. W. (2000). The intelligent essay assessor. IEEE Intelligent Systems & Their Applications, 15(5), 27-31.
25. Lee, I. (2014). Teachers’ reflection on implementation of innovative feedback approaches in EFL writing. English Teaching, 69(1), 23-40.
26. Mayfield, E. & Rose, C. P. (2013). LightSIDE: Open Source Machine Learning for Text. In M.D. Shermis& J.C. Burstein (Eds.), Handbook of automated essay evaluation: Current application and new directions (pp. 124-135). New York: Psychology Press.
27. McMillan, J. H. (2007). Classroom assessment: Principles and practice for effective standards-based instruction. Boston: Pearson.
28. Norbert, E. & Williamson, D. M. (2013). Assessing writing special issue: Assessing writing with automated scoring systems. Assessing Writing, 18(1), 1-6.
29. Nunan, D. (2010). Technology Supports for Second Language Learning. In P. Peterson, E. Baker, & B. McGaw (Eds.), International Encyclopedia of Education (pp. 204-210). Elsevier.
30. Page, E. B. (2003). Project essay grade: PEG. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 43-54). Mahwah, NJ: Lawrence Erlbaum Associates.
31. Paran, A. (2006). The stories of literature and language teaching. In A. Paran (Ed.), Literature in language teaching and learning (pp. 1-10). Alexandria, VA: TESOL.
32. Powers, D. E., Escoffery, D. S., & Duchnowski, M. P. (2015) Validating automated essay scoring: A (modest) refinement of the “gold standard”. Applied Measurement in Education, 28(2), 130-142. doi: 10.1080/08957347.2014.1002920
33. Ritter, K. (2012). Ladies who don’t know us correct our papers: Postwar lay reader programs and twenty-first century contingent labor in first-year writing. College Composition and Communication, 63(3), 387-419.
34. Sakiyama, Y., Yuki, H., Moriya, T., Hattori, K., Suzuki, M., Shimada, K., & Honma, T. (2008) Predicting human liver microsomal stability with machine learning techniques. J Mol Graph Model, 26, 907-915
35. Santos, V. D. O., Verspoor, M., & Nerbonne, J. (2012). Identifying important factors in essay grading using machine learning. In D. Tsagari, S. Papadima-Sophocleous, & S. Ioannou-Georgiou (Eds.), International experiences in language testing and assessment—Selected papers in memory of Pavlos Pavlou (pp. 295–309). Frankfurt am Main, Germany: Peter Lang GmbH.
36. Shermis, M. D. & Burstein, J. (2003). Automated essay scoring: A cross disciplinary perspective. Mahwah, NJ: Lawrence Erlbaum Associates.
37. Shermis, M. D., & Di Vesta, F. J. (2011). Classroom assessment in action. Maryland: Rowman and Little Field Publishers.
38. Shermis, M. D. & Hamner, B. (2013). Contrasting state-of-the-art automated scoring of essays. In M. D. Shermis & J. Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new directions (pp. 213–246). New York, NY: Routledge.
39. Stankova, E. N., Balakshiy, A. V., Petrov, D. A., Shorov, A. V., & Korkhov, V. V. (2016). Using technologies of OLAP and machine learning for validation of the numerical models of convective clouds. Lecture Notes in Computer Science, 9788, 463-472. doi: 10.1007/ 978-3-319-42111-7_36
40. Trivedi, S., Pardos, Z. A., & Heffernan, N. T. (2015). The utility of clustering in prediction tasks. CoRR, 1509.06163. Retrieved on 01 October 2018 from https://arxiv.org/abs/1509. 06163
41. Viera, A. J. & Garrett, J. M. (2005). Understanding interobserver agreement: the kappa statistic. Fam Med, 37(5), 360-363.
42. Weigle, S. C. (2013). English language learners and automated scoring of essays: Critical considerations. Assessing Writing, 18(1), 85-99.
43. Westera, W., Dascalu, M., Kurvers, H., Ruseti, S., & Trausan-Matu, S. (2018). Automated essay scoring in applied games: Reducing the teacher bandwidth problem in online training. Computers & Education, 123, 212-224. https://doi.org/10.1016/j.compedu.2018.05. 010
44. Wilson, J. (2018). Universal screening with automated essay scoring: Evaluating classification accuracy in grades 3 and 4. Journal of School Psychology, 68, 19-37. https://doi.org/ 10.1016/j.jsp.2017.12.005
45. Yamamoto, M., Umemura, N., & Kawano, H. (2018). Automated essay scoring system based on rubric. In R. Lee (Ed.), Applied computing & information technology. ACIT 2017. Studies in computational intelligence, vol. 727 (pp. 177-190). Springer, Cham. https://doi.org/ 10.1007/978-3-319-64051-8_11
46. Yang, M., Kim, M., Lee, H., & Rim, H. (2012). Assessing writing fluency of non-English speaking student for automated essay scoring – How to automatically evaluate the fluency in English essay. Paper presented at the 4th International Conference on Computer Supported Education. Porto, Portugal. Retrieved on 11 March 2016 from http://www. researchgate.net
47. Yang, W. (2012). A study of students’ perceptions and attitudes towards genre-based ESP writing instruction. The Asian ESP Journal, 8(3), 50-73.
48. Young, V. M., & Kim, D. H. (2010). Using assessments for instructional improvement: A literature review. Education Policy Analysis Archives, 18(19), 1-40.