Does a Formal Post-editing Training Affect the Performance of Novice Post-editors? An Experimental Study

Abstract

Machine translation systems led to the creation of a new role for translators: the post-editor. With the birth of neural machine translation systems, the demand for post-editing has been increasing in the recent years, and it has now become a common service given by language service providers and professional translators. Such a change in the landscape of the translation industry might evolve the translation training programs worldwide. It is still heavily discussed whether post-editing and translation skills overlap, and post-editing courses are now included into the curriculum by several translation departments. We set out to investigate whether post-editing training influences the performance of student post-editors in order to explore the necessary background and skills in post-editing tasks. We measured productivity parameters and quality of the final outputs produced by two groups of participants, one of which was previously trained on post-editing. Our results show that, the experimental and control groups did not differ significantly from each other in terms of productivity. There was also little to no difference when we evaluated the post-edited outputs produced by both groups against a reference text using automatic machine translation evaluation metrics. However, we detected a statistical significance between the groups when we analyzed the number of errors in the final output. The post-editors in the experimental group were more aware of the typical errors of machine translation engines.

Keywords

• machine translation
• post-editing
• translator training
• translation curriculum

Post-editing Eğitimi, Acemi Post-editörlerin Performansını Etkiliyor mu? Deneysel Bir Çalışma

Abstract

Makine çevirisi sistemleri, çevirmenler için yeni bir rolün oluşumuna yol açmıştır: post-editör. Nöral makine çevirisi sistemlerinin doğuşuyla post-editing hizmeti için talep son yıllarda artmaktadır ve artık dil hizmeti sağlayıcıları ile profesyonel çevirmenler tarafından sağlanan yaygın bir hizmet haline gelmiştir. Çeviri endüstrisindeki bu değişim, dünya genelindeki çeviri eğitimi programlarında köklü bir değişime yol açabilir. Post-editing ve çeviri becerilerinin birbiriyle ne ölçüde benzeştiği hâlâ tartışmalıdır ve bazı çeviri departmanlarının müfredatına post-editing dersleri eklenmiştir. Bu çalışmada, post-editing projelerinde gerekli arka planı ve becerileri incelemek için post-editing eğitiminin öğrenci post-editörlerin performansını etkileyip etkilemediği araştırılmıştır. Biri post-editing konusunda eğitilen iki katılımcı grubunun sunduğu nihai çıktıların kalitesi ve üretkenlik parametreleri ölçülmüştür. Sonuçlar, deney ve control gruplarının üretkenlik bakımından birbirinden anlamlı şekilde farklı olmadığını göstermiştir. Post-editing uygulanan çıktılar, otomatik makine çevirisi değerlendirme yöntemleri kullanılarak referans metinle karşılaştırıldığında da neredeyse hiç fark gözlenmemiştir. Fakat nihai çevirideki hata sayısı analiz edildiğinde gruplar arasında istatistiksel olarak anlamlı bir fark görülmüştür. Deney grubundaki post-editörler, makine çevirisi motorlarının tipik hatalarını daha kolay fark etmiştir.

Keywords

• makine çevirisi
• post-editing
• çevirmen eğitimi
• çeviri müfredatı

References

1. Allen, Jeffrey. “Post-Editing.” Computers and Translation: A Translator’s Guide, edited by Harold Somers, John Benjamins Publishing Company, 2003, pp. 297–318.
2. Aranberri, Nora. “What Do Professional Translators Do When Post-Editing for the First Time? First Insight into the Spanish-Basque Language Pair.” HERMES - Journal of Language and Communication in Business, no. 56, 2017, pp. 89–110, https://doi.org/10.7146/hjlcb.v0i56.97235.
3. Austermuehl, Frank. Future (and Not-so-Future) Trends in the Teaching of Translation Technology. http://revistes.uab.cat/tradumaticaElscontingutsdelarevistaestansubjectesaunallicènciaCreativeCommons. Accessed 23 Dec. 2018.
4. Balkul, Halil İbrahim. Türkiye’de Akademik Çeviri Eğitiminde Çeviri Teknolojilerinin Yerinin Sorgulanması: Müfredat Analizi ve Öğretim Elemanlarının Konuya Ilişkin Görüşleri Üzerinden Bir Inceleme. Sakarya University, 2015.
5. Clark, Jonathan H., et al. “Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability.” Acl-2011, 2011, pp. 176–81, https://doi.org/10.1057/dev.2008.5.
6. Daems, Joke, et al. “Translation Methods and Experience: A Comparative Analysis of Human Translation and Post-Editing with Students and Professional Translators.” Meta: Journal Des Traducteurs, vol. 62, no. 2, 2017, p. 245, https://doi.org/10.7202/1041023ar.
7. Denkowski, Michael, and Alon Lavie. “Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems.” Proceedings of the Sixth Workshop on Statistical Machine Translation, 2011, pp. 85–91, https://doi.org/10.1080/00288306.2004.9515087.
8. Depraetere, Ilse, et al. Post-Edited Quality, Post-Editing Behaviour and Human Evaluation: A Case Study. 2014, pp. 78-108., https://hal.archives-ouvertes.fr/halshs-01060447.

9. Diño, Gino. “Google, Facebook, Amazon: Neural Machine Translation Just Had Its Busiest Month Ever | Slator.” Slator, 2018, https://slator.com/technology/google-facebook-amazon-neural-machine-translation-just-had-its-busiest-month-ever/.
10. Garcia, Ignacio. “Is Machine Translation Ready Yet?” Target, vol. 22, no. 1, 2010, pp. 7–21, https://doi.org/10.1075/target.22.1.02gar.
11. Gaspari, Federico, et al. “A Survey of Machine Translation Competences: Insights for Translation Technology Educators and Practitioners.” Perspectives: Studies in Translatology, vol. 23, no. 3, 2015, pp. 333–58, https://doi.org/10.1080/0907676X.2014.979842.
12. Hutchins, W. John. Machine Translation: Past, Present, Future. Ellis Horwood; Halsted Press, 1986.
13. Kenny, Dorothy, and Stephen Doherty. Statistical Machine Translation in the Translation Curriculum: Overcoming Obstacles and Empowering Translators Statistical Machine Translation in the Translation Curriculum: Overcoming Obstacles and Empowering Translators. no. April 2018, 2014, https://doi.org/10.1080/1750399X.2014.936112.
14. Koponen, Maarit. “How to Teach Machine Translation Post-Editing? Experiences from a Post-Editing Course.” Proceedings of 4th Workshop on Post-Editing Technology and Practice (WPTP4), 2015.
15. Lardilleux, Adrien, et al. CHARCUT: Human-Targeted Character-Based MT Evaluation with Loose Differences To Cite This Version: HAL Id: Hal-01726326 C HAR C UT: Human-Targeted Character-Based MT Evaluation with Loose Differences. 2018.
16. Le, Quoc v., and Mike Schuster. “A Neural Network for Machine Translation, at Production Scale.” Google AI Blog, 27 Sept. 2016, https://ai.googleblog.com/2016/09/a-neural-network-for-machine.html.
17. Massardo, Isabella, et al. MT POST-EDITING GUIDELINES. TAUS Signature Editions, 2016.
18. O’Brien, Sharon. “Teaching Post-Editing: A Proposal for Course Content.” Proceedings of the 6th EAMT Workshop: Teaching Machine Translation, European Association for Machine Translation, 2002, https://aclanthology.org/2002.eamt-1.11.
19. Papineni, Kishore, et al. “BLEU: A Method for Automatic Evaluation of Machine Translation.” 40th Annual Meeting of the Association for Computational Linguistics, 2002, pp. 311–18, https://doi.org/10.1002/andp.19223712302.
20. Post-Editing Machine Translation Training. https://www.sdltrados.com/learning/training/post-editing-machine-translation.html. Accessed 24 Dec. 2018.
21. R Core Team. R: A Language and Environment for Statistical Computing. 2018, https://www.r-project.org/.
22. Rico, Celia, and Enrique Torrejón. “Skills and Profile of the New Role of the Translator as MT Post-Editor.” Revista Tradumàtica: Tecnologies de La Traducció, vol. 2012, no. 10, 2012, pp. 166–78, http://revistes.uab.cat/http://revistes.uab.cat/tradumatica.
23. Şahin, Mehmet. “Using MT Post-Editing for Translator Training.” Tralogy, vol. II, no. 6, 2011.
24. Six, Shawn E. Summary of the ATA Translation and Interpreting Services Survey | The Chronicle. 2014, http://www.atanet.org/chronicle-online/featured/summary-of-the-ata-translation-and-interpreting-services-survey/#sthash.6MTcHkO3.h9ME1ijM.dpbs.
25. Snover, Matthew, et al. “A Study of Translation Edit Rate with Targeted Human Annotation.” Proceedings of Association for Machine Translation in the Americas, 2006, pp. 223–31, https://doi.org/10.1.1.129.4369.
26. TAUS Research-Postediting in Practice. 2010, http://taus-website-media.s3.amazonaws.com/images/stories/pdf/benchmarch-data-for-postediting-practices-globally.pdf.
27. Temizöz, Özlem. “Postediting Machine Translation Output: Subject-Matter Experts versus Professional Translators.” Perspectives: Studies in Translatology, vol. 24, no. 4, 2016, pp. 646–65, https://doi.org/10.1080/0907676X.2015.1119862.
28. Thames, Jonathan. “Machine Translation.” LanguageSolutions, 24 June 2019, https://langsolinc.com/machine-translation.