A critical perspective on the translation quality assessments of five translators organizations: ATA, CTTIC, ITI, NAATI, and SATI

Abstract

The present paper discusses translation quality assessments by adopting a critical perspective on five translators organizations, which are intended to assess the quality of non-literary translations with a particular focus on their objectivity, validity, and inter-rater reliability. Within this framework, it aims to contribute to the related literature (1) by discussing the objectivity, validity, and inter-rater reliability of the quality assessment methods of five translators organizations, namely the American Translators Association (ATA), the South African Translators’ Institute (SATI), the Canadian Translators, Terminologists and Interpreters Council (CTTIC), the Institute of Translation and Interpreting (ITI in UK), and the National Accreditation Authority for Translators and Interpreters (NAATI in Australia), and (2) by making suggestions on quality assessment concerning non-literary translation in view of the findings hereof. The study qualitatively analyzes the content of the guides and rubrics provided on the websites of these five organizations and discusses the objectivity, validity, and inter-rater reliability of their assessments in consideration of seven parameters, i.e. “purpose of assessment”, “purpose of assigned translation”, “duration”, “source text”, “assessor”, “marking”, and “grading”. The findings showed that each organization suffers from varying degrees of objectivity, validity, and inter-rater reliability issues.

Keywords

• Translation quality assessment
• objectivity
• validity
• inter-rater reliability
• translators organizations

Beş çevirmenler derneğinin çeviri kalitesi ölçme uygulamalarına eleştirel bir bakış: ATA, CTTIC, ITI, NAATI ve SATI

Öz

Bu çalışma, beş çeviri örgütünün yazınsal olmayan çevirilerin kalitelerini değerlendirmeyi amaçlayan çeviri kalitesi değerlendirme uygulamalarını; nesnellik, geçerlilik ve değerbiçiciler arası güvenilirlik açısından eleştirel bir bakış açısıyla değerlendirmektedir. Bu bağlamda, (1) the American Translators Association (ATA), the South African Translators’ Institute (SATI), the Canadian Translators, Terminologists and Interpreters Council (CTTIC), the Institute of Translation and Interpreting (ITI - Birleşik Krallık), ve the National Accreditation Authority for Translators and Interpreters (NAATI - Avustralya)’dan oluşan beş örgütün çeviri kalitesi değerlendirme yöntemlerini nesnellik, geçerlilik, ve değerbiçiciler arası güvenilirliklerini tartışarak ve (2) bu çalışmanın bulguları çerçevesinde yazınsal olmayan çevirilerin kalitelerinin değerlendirilmesine değgin önerilerde bulunarak alanyazına katkı sağlamayı amaçlamaktadır. Çalışma; söz konusu örgütlerin internet sitelerinde verilen kılavuz ve standartların içeriklerini nitel olarak çözümlemekte ve “değerlendirmenin amacı”, “istenilen çevirinin amacı”, “süre”, “kaynak metin”, “değerlendirici”, “puanlama” ve “derecelendirme”den oluşan yedi değişkeni göz önünde bulundurarak bu beş örgütün nesnellik, geçerlilik ve değerbiçiciler arası güvenilirliklerini ele almaktadır. Bulgular, herbir örgütün değişen düzeylerde nesnellik, geçerlilik ve değerbiçiciler arası güvenilirlik sorunları olduğunu göstermiştir.

Anahtar Kelimeler

• Çeviri kalitesinin ölçülmesi
• nesnellik
• geçerlilik
• değerbiçiciler arası güvenilirlik
• çeviri örgütleri

References

1. “A Guide to the ATA Certification Program,” American Translator Association, accessed October 4, 2018, http://atanet.org/certification/aboutcert_overview.php.
2. “Accreditation Examinations: Guidelines on Marking,” South African Translators’ Institute, accessed October 4, 2018, http://translators.org.za/sati_cms/downloads/dynamic/majors_minors_english.pdf.
3. Al-Qinai, J. (2000). “Translation Quality Assessment. Strategies, Parametres and Procedures.” Meta: journal des traducteurs /Meta:Translators' Journal, 45(3): 497–519. https://doi.org/10.7202/001878ar
4. Angelelli, C. (2009). “Using a Rubric to Assess Translation Ability: Defining the Construct.” In Testing and Assessment in Translation and Interpreting Studies: A Call for Dialogue between Research and Practice, ed. by Claudia V. Angelelli and Holly E. Jacobson, 13-48. Amsterdam/Philadelphia: John Benjamins Publishing Company.
5. Angelelli, C. (2012). “Testing and Assessment in Translation and Interpreting Studies.” In Handbook of Translation Studies, ed. by Yves Gambier and Luc van Doorslaer, 172-177. Amsterdam/Philadelphia: John Benjamins Publishing Company.
6. “Applicant Handbook: Assessment Guide for Translators,” Institute of Translation and Interpreting, accessed October 4, 2018, https://www.iti.org.uk/attachments/article/992/ITI%20Applicant%20Handbook%20v4.pdf.
7. Ary, D; Lucy C. Jacobs ; Sorensen, C. (2010). Introduction to Research in Education. California: Wadsworth.
8. “ATA Certification Program Framework for Standardized Error Marking - Version 2017,” Last updated January 27, 2017, American Translator Association, accessed October 4, 2018, https://www.atanet.org/certification/Framework_2017.pdf.

9. Bittner, H. (2011). “The Quality of Translation in Subtitling.” Trans-kom, 4 (1): 76-87. http://www.trans-kom.eu/bd04nr01/trans-kom_04_01_04_Bittner_Quality.20110614.pdf.
10. Bowker, L. (2000). “A Corpus-based Approach to Evaluating Student Translations.” The Translator 6(2): 183-210. https://doi.org/10.1080/13556509.2000.10799065.
11. Castilho, S. ; Doherty, s. ; Gaspari, F. ; Moorkens, J. (2018). “Approaches to Human and Machine Translation Quality Assessment.” In Translation Quality Assessment: From Principles to Practice, ed. by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 9-38. Cham: Springer International Publishing.
12. “Certified Translator Test: Instructions for Candidates,” Last updated October 4, 2018, National Accreditation Authority for Translators and Interpreters, accessed October 5, 2018, https://www.naati.com.au/media/1951/ct-candidate-instructions-finalpdf.pdf.
13. Cohen, L. ; Manion, L. ; Morrison, K. (2000). Research Methods in Education. London: Routledge.
14. “Computerized Certification Exam,” American Translator Association, accessed October 4, 2018, https://atanet.org/certification/aboutexams_computerized.php.
15. “CTTIC Candidate’s Guide for the CTTIC Standard Certification Examination in Translation,” Last updated March, 2005, Canadian Translators, Terminologists and Interpreters Council, accessed October 4, 2018, http://www.cttic.org/examDocs/CandidateGuide_En1011.pdf
16. “CTTIC Standard Certification Translation Examination: Marker’s Guide,” Last updated March, 2005, Canadian Translators, Terminologists and Interpreters Council, accessed October 4, 2018, http://www.cttic.org/examDocs/guide.markersE.pdf.
17. English Oxford Living Dictionaries, s.v. “quality,” accessed October 07, 2018, https://en.oxforddictionaries.com/definition/quality.
18. “Flowchart for Error Point Decisions - Version 2009,” American Translator Association, accessed October 21, 2018, http://atanet.org/certification/aboutexams_flowchart.pdf.
19. “Framework for Marking of Translation Accreditation Examinations,” South African Translators’ Institute, accessed October 4, 2018, http://translators.org.za/sati_cms/downloads/dynamic/marking_guidelines_translation_english.pdf.
20. Hansen, G. (2003). “Controlling the Process: Theoretical and Methodological Reflections on Research into Translation Processes.” In Triangulating Translation: Perspectives in Process Oriented Research, ed. by Fabio Alves, 25-42. Amsterdam/Philadelphia: John Benjamins Publishing Company.
21. Hansen, G. (2008). “The Speck in Your Brother’s Eye – The Beam in Your Own: Quality Management in Translation and Revision.” In Efforts and Models in Interpreting and Translation Research, ed. by Gyde Hansen, Andrew Chesterman, and Heidrun Gerzymisch-Arbogast, 255-280. Amsterdam/Philadelphia: John Benjamins Publishing Company.
22. Hatim, B.; Mason, I. (2005). The Translator as Communicator. London/New York: Routledge.
23. Holmes, J. S. (2000). “The Name and Nature of Translation Studies.” In Translation Studies Reader, ed. by Lawrence Venuti, 172-185. London/New York: Routledge.
24. House, J. (1997). Translation Quality Assessment: A Model Revisited. Tübingen: Gunter Narr Verlag.
25. House, J. 2001. "Translation Quality Assessment: Linguistic Description versus Social Evaluation."Meta: Journal des Traducteurs/Meta: Translators' Journal 46 (2): 243-257. doi:10.7202/003141ar.
26. House, J. 2009. “Quality.” In Routledge Encyclopedia of Translation Studies, ed. by Mona Baker and Gabriela Saldanha, 222–225. London/New York: Routledge. “Information Regarding the Accreditation Examinations,” South African Translators’ Institute, accessed October 4, 2018, http://translators.org.za/sati_cms/downloads/dynamic/sati_accreditation_general_guidelines_english.pdf.
27. Koby, G. S. ; Champe, G. G. (2013). "Welcome to the Real World: Professional-Level Translator Certification."The International Journal for Translation and Interpreting Research 5(1): 156-173. doi:ti.105201.2013.a09.
28. Koby, G. S. ; Fields, P. ; Daryl R. Hague, Arle Lommel, and Alan Melby. 2014. “Defining Translation Quality.” Revista Tradumàtica: Tecnologies de la Traducció 12: 413-420. https://ddd.uab.cat/pub/tradumatica/tradumatica_a2014n12/tradumatica_a2014n12p413.pdf.
29. Krein-Kutiile, M. (2003) “Equivalence in Scientific and Technical Translation: A Text-in-Context-based Study.” PhD Diss., European Studies Research Institute, University of Salford.
30. Lommel, A. ; Popović, M. ; Burchardt, A. (2014). “Assessing Inter-Annotator Agreement for Translation Error Annotation.” In Proceedings of MTE Workshop on Automatic and Manual Metrics for Operational Translation Evaluation, LREC 2014, Reykjavik, Iceland.
31. Marsick, V. J. ; Watkins, K. E. (1990). Informal and Incidental Learning in the Workplace. London/New York: Routledge
32. Martínez-Melis, N., Hurtado Albir, A. (2001). “Assessment in Translation Studies: Research Needs. Meta:Journal des Traducteurs/Meta: Translators' Journal 46(2): 272–287. https://doi.org/10.7202/003624ar.
33. Morin, K. H. ; Barbin, F. ; Moreau, F. ; Toudic, D. ; Phuez-Favris, G. (2017). “Translation technology and learner performance: Professionally-oriented translation quality assessment with three translation technologies”. In Translation in Transition: Between Cognition, Computing and Technology, ed. by Arnt Lykke Jakobsen and Bartolomé Mesa-Lao, 207-233. Amsterdam/Philadelphia: John Benjamins Publishing Company.
34. “Policy for the Use of Equipment and Reference Materials in Certified Translator Tests and Certified Advanced Translator Tests,” Last updated March 8, 2018, National Accreditation Authority for Translators and Interpreters, accessed October 5, 2018, https://www.naati.com.au/media/1952/policy-use-equipment-reference-materials-trans-testspdf.pdf.
35. “Procedure to Appeal Certification Exam Review,” Last updated November 2009, American Translator Association, accessed February 12, 2019, http://atanet.org/certification/aboutreview_appeal.php
36. Rothe-Neves, R. (2002) . "Translation Quality Assessment for Research Purposes: An Empirical Approach.” Cadernos De Tradução 2(10): 113-131.https://doi.org/10.5007/%25x.
37. Schäffner, C. (1997). “From ‘Good’ to ‘Functionally Appropriate’: Assessing Translation Quality.” Current Issues in Language and Society 4(1). https://doi.org/10.1080/13520529709615476.
38. Schrijver, I. ; Van Vaerenbergh, L. (2012). “An Exploratory Study of Transediting in Students’ Translation Processes”. HERMES - Journal of Language and Communication in Business 25(49): 99-117. https://doi.org/10.7146/hjlcb.v25i49.97740.
39. Thelen, M. (2008). “Translation Quality Assessment or Quality Management and Quality Control of Translation?.” In Translation and Meaning – Part 8, ed. by Barbara Lewandowska-Tomaszczyk and Marcel Thelen, 411- 424. Maastricht: Hogeschool Zuyd.
40. Triandis, H. C. (1983). Allocentric vs. idiocentric social behavior: A major cultural difference between Hispanics and the mainstream. Technical Report No. 16. Department of Psychology, Urbana-Champaign: University of Illinois.
41. Williams, M. (1989). “The Assessment of Professional Translation Quality: Creating Credibility out of Chaos.” TTR: Traduction, Terminologie, Rédaction 2(2): 13-33. https://doi.org/10.7202/037044ar.
42. Williams, M. (2009). “Translation Quality Assessment.” Mutatis Mutandis 2(1): 3-23. http://aprendeenlinea.udea.edu.co/revistas/index.php/mutatismutandis/article/view/1825/1609.
43. Yazıcı, M. (2007). Yazılı Çeviri Edinci. İstanbul: Multilingual
44. Zehnalová, J. (2013). “Tradition and Trends in Translation Quality Assessment”.In Tradition and Trends in Trans-Language Communication, ed. by Jitka Zehnalová, Ondřej Molnár, and Michal Kubánek, 41-58. Olomouc: Univerzita Palackého.