Post-Editing Oriented Human Quality Evaluation of Neural Machine Translation in Translator Training: A Study on Perceived Difficulties and Benefits

Abstract

The aim of this article is to investigate translation trainees’ perceived difficulties and benefits of a post-editing oriented neural machine translation (NMT) error annotation and quality evaluation task which was carried out for the language pair English-Turkish and for two separate text types and domains, i.e., environmental blogs and movie/TV reviews, within the scope of an MA course on translation quality standards. The data to be analyzed were collected through a semi-structured questionnaire which was given to the trainees attending the course after the completion of the task. The questionnaire was prepared with the aim of understanding perceived difficulties and benefits of the task. Analysis of the answers revealed that most of the trainees were in the opinion that the task was difficult. Majority of the trainees also believed that the task was beneficial and enabled them to feel empowered to make decisions on human translation quality evaluation of machine translation (MT) and to carry out error annotation and post-editing activities in the future. According to a significant number of trainees, error annotation facilitated their post-editing process and reduced the effort in post-editing. Enhanced understanding of MT error annotation and enhanced ability to perform post-editing were the significant benefits stated by the trainees. The difficulties were associated with being introduced and assigned to perform tasks they were not familiar with. Yet, as displayed in the answers to the questionnaire, a considerable majority of the trainees were positive about the learning experience. The results have also shown that integration of MT-related activities into translator training with a focus on the empowerment of the human translator has its difficulties and benefits also for translation trainers. While the difficulties for the trainers concern the decisions on the design, implementation, and planning of the task and the responsibility to carry out the task in constant interaction and collaboration with the trainees, the benefits are the sense of fulfillment and enrichment brought by positive feedback from the trainees and the discovery of the fact that the so-called ‘teaching experience’ becomes a ‘learning experience’ for trainers as well as trainees.

Keywords

• perceived difficulties and benefits
• human translator
• neural machine translation
• human quality evaluation
• error annotation
• post-editing
• translation quality standards

References

1. Alimen, Nilüfer, and Senem Öner Bulut. 2020. “Çevirinin Teknolojikleşmesi Bağlamında İnsan Çevirmenin Rollerini Yeniden Düşünmek: Çevirmen Eğitiminde Teknik Metin Yazarlığı.” [Re-thinking the roles of the human translator in the context of the technologization of translation: Technical writing in translator training.] Turkish Studies – Language and Literature 15 (3): 1047–1062. doi:10.47845/TurkishStudies.45679.
2. Cadwell, Patrick, Sharon O’Brien, and Carlos S. C. Teixeira. 2018. “Resistance and Accommodation: Factors for the (Non-) Adoption of Machine Translation Among Professional Translators.” Perspectives 26 (3): 301–321. doi:10.1080/0907676X.2017.1337210.
3. Castilho, Sheila. 2020. “On the Same Page? Comparing Inter-Annotator Agreement in Sentence and Document Level Human Machine Translation Evaluation.” In Proceedings of the 5th Conference on Machine Translation (WMT), 1150–1159. Association for Computational Linguistics (ACL). https://www.aclweb.org/anthology/2020.wmt-1.137.pdf.
4. Castilho, Sheila, Joss Moorkens, Federico Gaspari, Rico Sennrich, Andy Way, and Panayota Georgakopoulou. 2018. “Evaluating MT for Massive Open Online Courses: A Multifaceted Comparison Between PBSMT and NMT Systems.” In “Human Evaluation of Statistical and Neural Machine Translation,” edited by Andy Way and Mikel L. Forcada. Special Issue, Machine Translation 32 (3): 255–278. doi:10.1007/s10590-018-9221-y.
5. Doherty, Stephen, and Dorothy Kenny. 2014. “The Design and Evaluation of a Statistical Machine Translation Syllabus for Translation Students.” The Interpreter and Translator Trainer 8 (2): 295–315. doi:10.1080/1750399X.2014.937571.
6. Freitag, Markus, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. “Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation.” arXiv preprint arXiv:2104.14478, April 29. https://arxiv.org/pdf/2104.14478.pdf.
7. Garcia, Ignacio. 2011. “Translating by Post-Editing: Is It the Way Forward?” Machine Translation 25 (3): 217–237. doi:10.1007/s10590-011-9115-8.
8. Gaspari, Federico, Hala Almaghout, and Stephen Doherty. 2015. “A Survey of Machine Translation Competences: Insights for Translation Technology Educators and Practitioners.” Perspectives 23 (3): 333–358. doi:10.1080/0907676X.2014.979842.

9. Guerberof Arenas, Ana. 2013. “What Do Professional Translators Think About Post-Editing?” In “Machine Translation and the Working Methods of Translators,” edited by Louise Brunette. Special Issue, JoSTrans The Journal of Specialised Translation, no. 19, 75–95. https://www.jostrans.org/issue19/art_guerberof.pdf.
10. Haro-Soler, Maria del Mar, and Don Kiraly. 2019. “Exploring Self-Efficacy Beliefs in Symbiotic Collaboration with Students: An Action Research Project.” In “Training the Trainers,” edited by Gary Massey, Don Kiraly, and Maureen Ehrensberger-Dow. Special Issue, The Interpreter and Translator Trainer 13 (3): 255–270. doi:10.1080/1750399X.2019.1656405.
11. Hiraoka, Yusuke, and Masaru Yamada. 2019. “Pre-Editing Plus Neural Machine Translation for Subtitling: Effective Pre-Editing Rules for Subtitling of TED Talks.” In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, 64–72. https://www.aclweb.org/anthology/W19-6710.pdf.
12. Kenny, Dorothy, and Stephen Doherty. 2014. “Statistical Machine Translation in the Translation Curriculum: Overcoming Obstacles and Empowering Translators.” The Interpreter and Translator Trainer 8 (2): 276–294. doi:10.1080/1750399X.2014.936112.
13. Kiraly, Don. 2000. A Social Constructivist Approach to Translator Education: Empowerment from Theory to Practice. Manchester: St. Jerome.
14. Klubička Filip, Antonio Toral, and Víctor M. Sánchez-Cartagena. 2017. “Fine-Grained Human Evaluation of Neural Versus Phrase-Based Machine Translation.” The Prague Bulletin of Mathematical Linguistics, no. 108, 121–132. doi:10.1515/pralin-2017-0014.
15. Klubička Filip, Antonio Toral, and Víctor M. Sánchez-Cartagena. 2018. “Quantitative Fine-Grained Human Evaluation of Machine Translation Systems: A Case Study on English to Croatian.” In “Human Evaluation of Statistical and Neural Machine Translation,” edited by Andy Way and Mikel L. Forcada. Special Issue, Machine Translation 32 (3): 195–215. doi:10.1007/s10590-018-9214-x.
16. Koponen, Maarit. 2016. “Is Machine Translation Post-Editing Worth the Effort? A Survey of Research into Post-Editing and Effort.” In “The Translation Profession: Centres and Peripheries,” edited by Helle V. Dam and Kaisa Koskinen. Special Issue, JoSTrans The Journal of Specialised Translation, no. 25, 131–148. https://www.jostrans.org/issue25/art_koponen.pdf.
17. Mellinger, Christopher D. 2017. “Translators and Machine Translation: Knowledge and Skills Gaps in Translator Pedagogy.” The Interpreter and Translator Trainer 11 (4): 280–293. doi:10.1080/1750399X.2017.1359760.
18. Mercader-Alarcón, Julia, and Felipe Sánchez-Martínez. 2016. “Analysis of Translation Errors and Evaluation of Pre-Editing Rules for the Translation of English News Texts into Spanish with Lucy LT.” Traducció i dispositius mòbils Revista Tradumàtica: tecnologies de la traducció, no. 14, 172–186. doi:10.5565/rev/tradumatica.164.
19. O’Brien, Sharon. 2002. “Teaching Post-Editing: A Proposal for Course Content.” In Proceedings of 6th EAMT Workshop: Teaching Machine Translation, 99–106. https://www.aclweb.org/anthology/2002.eamt-1.11.pdf.
20. O’Brien, Sharon. 2006. “Machine-Translatability and Post-Editing Effort: An Empirical Study Using Translog and Choice Network Analysis.” PhD diss., Dublin City University.
21. O’Brien, Sharon. 2012. “Translation as Human–Computer Interaction.” Translation Spaces 1 (1): 101–122. doi:10.1075/ts.1.05obr.
22. O’Brien, Sharon, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia, eds. 2014. Post-Editing of Machine Translation: Processes and Applications. Newcastle upon Tyne: Cambridge Scholars.
23. Olohan, Maeve. 2011. “Translators and Translation Technology: The Dance of Agency.” Translation Studies 4 (3): 342–357. doi:10.1080/14781700.2011.589656.
24. Öner, Işın. 2018. “Why Technical Writing is One of the Basic Courses in Translation Studies.” Intelligent Information Blog. January 31. https://intelligent-information.blog/en/why-technical-writing-is-one-of-the-basic-courses-in-translation-studies.
25. Öner, Işın. 2019. “Teknik Metin Yazarlığı Nedir? Çeviri Eğitimindeki Yeri Nedir?” [What is technical writing? What is its position in translation training?] Speech delivered at the İstanbul Üniversitesi Edebiyat Fakültesi Çeviribilim Bölümü Çeviri Söyleşileri 16 [Istanbul University Faculty of Letters Department of Translation Studies Conversations on Translation 16], March 1.
26. Öner, Işın, and Senem Öner. 2011. “Exploring the Writing and Translation of Technical Documentation for Its Integration into Translator Training Programs in Turkey: A Preliminary Model.” Paper presented at the Aspects in Production and Translation of Technical Documentation International Tekom Conference, Istanbul, December 12.
27. Öner Bulut, Senem. 2019a. “Future Professional Profile and Agency of the Human Translator: A Survey on Human-Machine Tension in the Context of the Technologization of Translation.” In Çeviribilimde Araştırmalar [Research in Translation Studies], edited by Seda Taş, 93–122. Istanbul: Hiperyayın.
28. Öner Bulut, Senem. 2019b. “Integrating Machine Translation into Translator Training: Towards ‘Human Translator Competence’?” transLogos Translation Studies Journal 2 (2): 1–26. doi:10.29228/transLogos.11.
29. Pym, Anthony. 2019. “How Automation Through Neural Machine Translation Might Change the Skill Sets of Translators.” Accessed September 10, 2019. https://www.academia.edu/40200406/How_automation_through_neural_machine_tran%20slation_might_change_the_skill_sets_of_translators.
30. Ruokonen, Minna, and Kaisa Koskinen. 2017. “Dancing with Technology: Translators’ Narratives on the Dance of Human and Machinic Agency in Translation Work.” The Translator 23 (3): 310–323. doi:10.1080/13556509.2017.1301846.
31. Sakamoto, Akiko. 2019. “Unintended Consequences of Translation Technologies: From Project Managers’ Perspectives.” Perspectives 27 (1): 58–73. doi:10.1080/0907676X.2018.1473452.
32. Seljan, Sanja, Ivan Dunđer, and Marko Pavlovski. 2020. “Human Quality Evaluation of Machine-Translated Poetry.” In 2020 43rd International Convention on Information, Communication and Electronic Technology (MIPRO), 1040–1045. doi:10.23919/MIPRO48935.2020.9245436.
33. Shei, Chi-Chiang. 2002. “Teaching MT Through Pre-Editing: Three Case Studies.” In Proceedings of the 6th EAMT Workshop: Teaching Machine Translation. https://www.aclweb.org/anthology/2002.eamt-1.10.pdf.
34. Somers, Harold. 1997. “A Practical Approach to Using Machine Translation Software: ‘Post-Editing’ the Source Text.” The Translator 3 (2): 193–212. doi:10.1080/13556509.1997.10798998.
35. Vieira, Lucas Nunes. 2019. “Post-Editing of Machine Translation.” In The Routledge Handbook of Translation and Technology, edited by Minako O’Hagan, 319–335. London: Routledge.
36. Vieira, Lucas Nunes. 2020. “Automation Anxiety and Translators.” Translation Studies 13 (1): 1–21. doi:10.1080/14781700.2018.1543613.
37. Yuste, Elia. 2005. “Computer-Aided Technical Translation Workflows – Man-Machine in the Construction and Transfer of Corporate Knowledge.” Linguistik Online 23 (2): 67–75. doi:10.13092/lo.23.647.