AUTOMATED WRITING EVALUATION SYSTEM FOR FEEDBACK IN THE DIGITAL WORLD: AN ONLINE LEARNING OPPORTUNITY FOR ENGLISH AS A FOREIGN LANGUAGE STUDENTS

Year 2024, Volume: 25 Issue: 3, 183 - 206, 01.07.2024

Hilal Yildiz S. Ipek Kuru Gonen

https://doi.org/10.17718/tojde.1169727

Abstract

It is imperative to use new technologies in a supportive manner to meet the learners’ and teachers’ demanding needs as educational environments change in the digital age. The continuous expansion of online learning and distance education opportunities responds to the demands of learners and teachers while pioneering the use of technology in education. One advancement in English language teaching and learning in online environments, which assists teachers in reducing their workload and providing students with instant digital feedback, is the automated writing evaluation (AWE) tools. To gain a deeper understanding of the potential and limitations of these digital tools, this study aims to investigate the effectiveness of AWE feedback in error reduction in writing in English and the explore views of students regarding the utility of AWE tools. For this purpose, a total of 38 students at a university in Turkiye participated in the study, and three of their essays were evaluated. Within a concurrent triangulation mixed-method design, the changes in errors of the experimental group (n=18) receiving AWE feedback, and the control group (n=20) receiving teacher feedback were analyzed quantitatively, and the written reflection reports and semi-structured interviews conducted with the students were analyzed qualitatively. The results indicated that teacher feedback and AWE feedback were both effective in reducing errors in 11 categories. AWE feedback appeared to minimize errors in mechanics and usage more efficiently and teacher feedback was required more in content and organization issues. As a result, AWE was found as a complementary and effective tool supporting the improvement of target language writing skills saving time and energy for teachers. Furthermore, students expressed positive views regarding the use of AWE despite minor limitations. The findings of this study in general sheds light on using online digital tools of ubiquitous nature such as AWE to assist language improvement outside the class.

Keywords

Automated Writing Evaluation system for feedback, Online learning in L2 writing, online feedback in L2 writing, L2 writing evaluation in the distance learning environment, Students’ views on digital writing evaluation tool

References

• Amrhein, H. R., & Nassaji, H. (2010). Written corrective feedback: What do students and teachers think is right and why?. Canadian Journal of Applied Linguistics, 13(2), 95-127.
• Attali, Y. (2004). Exploring the feedback and revision features of Criterion. Journal of Second Language Writing, 14 (1), 191-205.
• Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® V. 2. The Journal of Technology, Learning and Assessment, 4(3), 1-31.
• Attali, Y., & Powers, D. (2008). Effect of immediate feedback and revision on psychometric properties of open-ended GRE Subject Test items: GRE Board Research Report. Princeton, NJ: ETS.
• Biber, D., Douglas, B., Conrad, S., & Reppen, R. (1998). Corpus linguistics: Investigating language structure and use. Cambridge: Cambridge University Press.
• Brown, G. A., Bull, J., & Pendlebury, M. (2013). Assessing student learning in higher education. New York: Routledge.
• Bulut, S. E. (2019). The effects of automated writing evaluation on EFL students' writing achievement and motivation towards writing. Unpublished Master’s Thesis. İstanbul: İstanbul University, Department of English Language Education.
• Calvo, R. A., & Ellis, R. A. (2010). Students' conceptions of tutor and automated feedback in professional writing. Journal of Engineering Education, 99(4), 427-438.
• Chapelle, C. A., Cotos, E., & Lee, J. (2015). Validity arguments for diagnostic assessment using automated writing evaluation. Language testing, 32(3), 385-405.
• Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. London: SAGE.
• Chaudron, C. (1988). Second language classrooms: Research on teaching and learning. Cambridge: Cambridge University Press.
• Chen, C. F. E., & Cheng, W. Y. E. C. (2008). Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in EFL writing classes. Language Learning & Technology, 12(2), 94-112.
• Cheng, G. (2017). The impact of online automated feedback on students' reflective journal writing in an EFL course. The Internet and Higher Education, 34(1), 18-27.
• Cotos, E. (2010). Automated writing evaluation for non-native speaker English academic writing: The case of IADE and its formative feedback. Unpublished Doctoral Dissertation. Iowa: Iowa State University, Faculty of Applied Linguistics and Technology.
• Cotos, E. (2014). Genre-based automated writing evaluation for L2 research writing: From design to evaluation and enhancement. New York: Springer.
• Deqi, Z. (2005). The process-oriented approach to ESL/EFL writing instruction and research. Celea Journal, 28(5), 66-70.
• Dikli, S. (2006). An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment, 5(1), 1–35.
• Dörnyei, Z. (2007). Research methods in applied linguistics. Oxford: Oxford.
• El Ebyary, K., & Windeatt, S. (2010). The impact of computer-based feedback on students’ written work. International Journal of English Studies, 10(2), 121-142.
• Elliott, S. (2003). IntellimetricTM: From here to validity. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 71-86). Mahwah, NJ: Lawrence Erlbaum Associates.
• Ferris, D. R. (2014). Responding to student writing: Teachers’ philosophies and practices. Assessing Writing, 19(1), 6–23.
• Ferris, D. R., & Roberts, B. (2001). Error feedback in L2 writing classes: How explicit does it need to be?. Journal of Second Language Writing, 10(3), 161-184.
• Galletta, A. (2013). Mastering the semi-structured interview and beyond: From research design to analysis and publication. New York. NYU press.
• Gençer, Ö. (2019) Automated writing evaluation use in an EFL context: From paragraph writing to essay writing. Unpublished Master’s Thesis. Ankara: Ufuk University, Institute of Social Sciences.
• Gilakjani, A. P. (2017). A review of the literature on the integration of technology into the learning and teaching of English language skills. International Journal of English Linguistics, 7(5), 95-106.
• Glaser, B., & Strauss, A., (1967). The discovery of grounded theory. Hawthorne, NY: Aldine Publishing Company.
• Gorgogeta, M., & Vlachos, K. (2019). A case study to compare the effects of different practices applied when assessing language learners. Research Papers in Language Teaching and Learning, 10(1), 241-264.
• Grimes, D. C. (2008). Middle school use of automated writing evaluation: A multi-site case study. Unpublished Doctoral dissertation. Irvine: University of California, Department of Information and Computer Science.
• Grimes, D. C., & Warschauer, M. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. The Journal of Technology, Learning and Assessment, 8(6), 4-43.
• Gurel, N. (2010). An examination of linguistic and sociocultural variables in writing a dissertation among Turkish doctoral students. Unpublished Doctoral Dissertation. (Publication. New York: State University of New York, Department of Learning and Instruction.
• Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112.
• Hockly, N. (2019). Automated writing evaluation. ELT Journal, 73(1), 82–88.
• Hsu, P. S. (2016). Examining current beliefs, practices and barriers about technology integration: A case study. TechTrends, 60(1), 30-40.
• Huang, M., Kuang, F., & Ling, Y. (2022). EFL learners’ engagement in different activities of blended learning environment. Asian-Pacific Journal of Second and Foreign Language Education, 7(1), 1-15.
• Jefferson, K., & Radhakrishnan, V. (2020). An Exploration on issues and challenges in teaching writing skills to tertiary-level learners. In S. Kumar and R. Mohanasundram, Innovations and technologies for soft skill development and learning (pp. 98-107). Hershey, PA: IGI Global.
• Jiang, L., Yu, S., & Wang, C. (2020). Second language writing instructors’ feedback practice in response to automated writing evaluation: A sociocultural perspective. System, 93, 102302.
• Joughin, G. (2009). Assessment, learning and judgement in higher education: A critical review. In G. Joughin (Ed.), Assessment, learning and judgement in higher education (pp. 1-15). New York, NY: Springer.
• Kamaşak, R., Sahan, K., & Rose, H. (2021). Academic language-related challenges at an English-medium university. Journal of English for Academic Purposes, 49(1), 1-40.
• Khoii, R., & Doroudian, A. (2013). Automated scoring of EFL learners’ written performance: A torture or a blessing. In Proceedings of the Edulearn14: 6th International Conference On Education And New Learning Technologies. Italy: International Education and Training Institution.
• Laurillard, D. (2013). Rethinking university teaching: A conversational framework for the effective use of learning technologies. New York: Routledge.
• Lee, I. (2003). L2 writing teachers’ perspectives, practices and problems regarding error feedback. Assessing Writing, 8(3), 216-237.
• Lerner, N. D. (1996). Teaching and learning in a university writing center. Unpublished dissertation, Massachusetts: Boston University, Department of Humanities and Social Sciences.
• Li, J., Link, S., & Hegelheimer, V. (2015). Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction. Journal of Second Language Writing, 27, 1-18.
• Limpo, T., Nunes, A., & Coelho, A. (2020). Introduction to the special issue on technology-based writing instruction: A collection of effective tools. Journal of Writing Research, 12(1), 1–7.
• Lombana, C. H. (2002). Some issues for the teaching of writing. Profile: Issues in Teachers´ Professional Development, 3(1), 44-51.
• Long, R. (2013). A review of ETS’s Criterion online writing program for student compositions. The Language Teacher, 37(3), 11-18.
• Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. London: SAGE.
• Mohsen, M. A., & Abdulaziz, A. (2019). The effectiveness of using a hybrid mode of automated writing evaluation system on EFL students’writing. Teaching English with Technology, 19(1), 118-131.
• Nordrum, L., Evans, K., & Gustafsson, M. (2013). Comparing student learning experiences of in-text commentary and rubric-articulated feedback: strategies for formative assessment. Assessment & Evaluation in Higher Education, 38(8), 919-940.
• Palermo, C., & Thomson, M. M. (2018). Teacher implementation of self-regulated strategy development with an automated writing evaluation system: Effects on the argumentative writing performance of middle school students. Contemporary Educational Psychology, 54(1), 255-270.
• Pintrich, P. R. (1999). The role of motivation in promoting and sustaining self-regulated learning. International journal of educational research, 31(6), 459-470.
• Prensky, M. (2001). Digital natives, Digital immigrants. On the Horizon, 9(5), 1-6.
• Ranalli, J., Link, S., & Chukharev-Hudilainen, E. (2017). Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation. Educational Psychology, 37(1), 8-25.
• Ruegg, R. (2010). Interlanguage development: The effect of unfocused feedback on L2 writing. Intercultural communication studies, 12(1), 247-254.
• Shermis, M. D., & Burstein, J. (2013). Handbook of automated essay evaluation: Current applications and new directions. New York: Routledge.
• Silva, T. (1993). Toward an understanding of the distinct nature of L2 writing: The ESL research and its implications. TESOL Quarterly, 27(4), 657-677.
• Stone, G. (2017). Implementation of critical literacy for English writing classes in the Thai context. The New English Teacher, 11(2), 65-65.
• Tawney, J., & Gast, D. (1984). Single-subject research in special education. New York: Merrill.
• Üstünbaş, Ü., & Çimen, S. (2016). EFL learners’ preferences for feedback types for their written products. The Online Journal of New Horizons in Education, 6(4), 68-74.
• Van Beuningen, C. (2010). Corrective feedback in L2 writing: Theoretical perspectives, empirical insights, and future directions. International Journal of English Studies, 10(2), 1-27.
• Wang, Y. J., Shang, H. F., & Briody, P. (2013). Exploring the impact of using automated writing evaluation in English as a foreign language university students' writing. Computer Assisted Language Learning, 26(3), 234-257.
• Ware, P. (2011). Computer-generated feedback on student writing. TESOL Quarterly, 45(4), 769-774.
• Warschauer, M., & Grimes, D. (2008). Automated writing assessment in the classroom. Pedagogies: An International Journal, 3(1), 22-36.
• Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2), 157-180.
• Weigle, S. C. (2013). English as a second language writing and automated essay evaluation. In M.D. Shermis, J.C. Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new directions, (pp. 36–54). New York: Routledge.
• Wilson, J., & Czik, A. (2016). Automated essay evaluation software in English Language Arts classrooms: Effects on teacher feedback, student motivation, and writing quality. Computers & Education, 100(1), 94-109.
• Wilson, J., & Roscoe, R. D. (2020). Automated writing evaluation and feedback: Multiple metrics of efficacy. Journal of Educational Computing Research, 58(1), 87-125.
• Zeng, S. (2020). The potential of online technology for language learning. English Language Teaching, 13(10), 23-37.
• Zhang, Z. (2020). Engaging with automated writing evaluation (AWE) feedback on L2 writing: Student perceptions and revisions. Elsevier Inc: Assessing Writing, 43, 1-14.
• Zhang, Z. V., & Hyland, K. (2018). Student engagement with teacher and automated feedback on L2 writing. Assessing Writing, 36(1), 90-102.
• Zhang, Z., & Zhang, Y. (2018). Automated writing evaluation system: Tapping its potential for learner engagement. IEEE Engineering Management Review, 46(3), 29-33.
• Zupanc, K., & Bosnic, Z. (2015). Advances in the field of automated essay evaluation. Informatica, 39(4), 383–395.