Çeviribilimde Çok Modlu Değişim: Sınırları Aşmak

Öz

Gelişen teknolojilerin artan etkileri ile bağlantılı olarak, modern çağda iletişim dilinin değişen yapısının farkına varılması üzerine çeviri araştırmasının kapsamı, anlamı oluşturmada dilbilimsel modun yanı sıra sözel olmayan biçimlerin de önemini benimseyecek bir şekilde yeniden tanımlanmıştır. Bu anlamda, çok modluluk kavramı, anlam aktarım sürecinin vazgeçilmez bir bileşeni olarak, gün geçtikçe daha da dikkat çekmiştir ve bu kavram ile Çeviribilim arasında bulunan çift yönlü bağlantı, söz konusu alanlardaki yakınsamanın net bir görüntüsünü yakalamak için yeni bir araştırma alanı olarak özellikle vurgulanmıştır. Bununla birlikte, ilgili alan yazında, özellikle çevirilerde çok modlu bütünlüğün işlevselliğine yönelik çalışmaların eksikliği söz konusudur. Bu durumu akılda tutarak, bu inceleme çalışması, önceki araştırmaların ışığında, (a) çok modluluk olgusunun çeviride kendisini nasıl konumlandırdığını, (b) çeviri araştırmasının kapsama alanını nasıl yapılandırdığını, (c) çok modlu metinlerin köklü çözümleme yöntemleri referans alınarak nasıl deşifre edildiğini ve (d) bu paradigmadaki çeviri araştırmasının ilerlediği mevcut yönleri belirlemeyi hedeflemektedir. Tüm bu amaçlar doğrultusunda, temel teşkil eden başlıca teorilere değinilerek, kavramsal konuların derinlemesine detaylandırılmasından başlayarak, bu makalenin yöntem bölümünde, çok modlu bir mesajın en iyi şekilde nasıl aktarılacağını belirlemek için farklı türlerde kullanılan bir takım teorik çerçeveler sunulmuştur. Genel olarak bu çalışma, çevirmenlerin metinlerin çok modlu katmanlaşması ve ilgili çeviri yöntemleri hakkında daha fazla farkındalık kazanmalarını sağlayarak bu alana yeni boyutlar kazandırabilir.

Anahtar Kelimeler

• çok modluluk
• görsel-işitsel çeviri
• mod
• sosyal göstergebilim
• sözlü olmayan modlar

Multimodal Change in Translation Studies: Transcending the Boundaries

Öz

Upon recognizing the changing landscape of the communication register in the modern era with the burgeoning impacts of emerging technologies, the scope of translation research has been redefined in a way that embraces the prominence of non-verbal modes, alongside the linguistic code in meaning-construction. In this sense, the multimodality concept has increasingly received attention as an indispensable component of the meaning-transfer process, and its bidirectional connection with Translation Studies has been specially emphasized as a new research avenue to capture a clear image of the convergence in these fields. However, there exists a dearth of studies in the related literature, specifically devoted to the functionality of multimodal cohesion in translations. With this in mind, in light of previous research, this review study seeks to determine (a) how the phenomenon of multimodality locates itself in translation, (b) how it structures the extent of translatology research, (c) how multimodal texts are deciphered with reference to well-established transcribing methods, and (d) the current directions that the trans lation research in this paradigm is moving towards. For all these purposes, starting with an in-depth elaboration of conceptual issues by addressing the underlying core theories, in the methodology section of this paper, a number of theoretical frameworks used in different genres were presented to specify how a multimodal message is best transferred. Overall, this study can add new dimensions to this discipline by enabling translators to gain more awareness of the multimodal stratification of texts and the relevant translation methods.

Anahtar Kelimeler

• audio-visual translation
• mode
• multimodality
• non-verbal modalities
• social semiotics

Kaynakça

1. Adami, E. & Ramos Pinto, S. (2020). Meaning-(re)making in a world of untranslated signs: towards a research agenda on multimodality, culture, and translation. In M. Boria, Á. Carreres, M. Noriega Sánchez, & M. Tomalin (Eds.), Translation and multimodality: Beyond words (pp. 71–93). Routledge: New York. google scholar
2. Akiba, Y., Federico, M., Kando, N., Nakaiwa, H., Paul, M., & Tsujii, J. (2004). Overview of the IWSLT 2004 evaluation campaign. In Proceedings of the 2004 international workshop on spoken language translation (IWSLT), Kyoto, pp. 1–12. google scholar
3. Boria, M., & Tomalin, M. (2020). Introduction. In M. Boria, Á. Carreres, M. Noriega Sánchez, & M. Tomalin (Eds.), Translation and multimodality: Beyond words (pp. 1–23). Routledge: New York. google scholar
4. Carreres, Á. & Noriega-Sánchez, M. (2020). Beyond words: concluding remarks. In M. Boria, Á. Carreres, M. Noriega-Sánchez, & M. Tomalin, (Eds), Translation and Multimodality. Beyond Words (pp. 198-203), Routledge: New York. google scholar
5. Chen, X. (2023). Representing cityscape through texts and images: translations of multimodal public notices in Macao, Asia Pacific Translation and Intercultural Studies, 10(1), 53-70. google scholar
6. Chen, Y. & Wang, W. (2016). Relating visual images to subtitle translation in Finding Nemo: A multi-semiotic interplay. Translation & Interpreting, 8(1), 69-85. google scholar
7. Çağlayan, O. (2019). Multimodal machine translation. Computation and Language (Unpublished PhD Dissertation). Le Mans Université, France. google scholar
8. Damaskinidis, G. (2016). The visual aspect of translation training in multimodal texts. Meta, 61(2), 299–319. https://doi.org/10.7202/ 1037761ar google scholar

9. Elliott, D., Frank, S., Sima’an, K., & Specia. L. (2016). Multi30K: Multilingual English- German image descriptions. In Proceedings of the 5th Workshop on Vision and Language (pp. 70– 74), Berlin, Germany. Association for Computational Linguistics. google scholar
10. Everett, Y. U. (2015). Reconfiguring myth and narrative in Contemporary opera. Bloomington: Indiana University Press. google scholar
11. Gambier, Y. (2006). Multimodality and Audiovisual Translation. Proceedings of the Marie Curie Euroconferences MuTra: Audiovisual translation scenarios (pp. 1-8). Retrieved from http://euroconferences.info/proceedings/2006_Proceedings/2006_Gambier_Yves. pdf google scholar
12. Gambier, Y. (2023). Audiovisual translation and multimodality: What future? Media and Intercultural Communication: A Multidisciplinary Journal, 1(1), 1-16. https://doi.org/10.22034/mic.2023.167451 google scholar
13. Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Arnold. google scholar
14. Hodge, R. & Kress, G. (1988). Social semiotics. Ithaca, NY: Cornell University Press. google scholar
15. Jakobson, R. (1959). On linguistic aspects of translation. In L. Venuti (Ed.), The Translation Studies Reader (pp. 232–239). London: Routledge. google scholar
16. Jewitt, C., Bezemer, J., & O’Halloran, K. L. (2016). Introducing multimodality. London: Routledge. google scholar
17. Kaindl, K. (2013). Multimodality and translation. In C. Millán & F. Bartrina (Eds.), The Routledge Handbook of Translation Studies (pp. 257–269). London: Routledge. google scholar
18. Kaindl, K. (2020). A theoretical framework for a multimodal conception of translation. In M. Boria, Á. Carreres, M. Noriega-Sánchez, & M. google scholar
19. Tomalin, (Eds), Translation and Multimodality. Beyond Words (pp. 49-70), Routledge: New York. google scholar
20. Ketola, A. (2016a). Towards a multimodally oriented theory of translation: A cognitive framework for the translation of illustrated technical texts. Translation Studies, 9(1), 67–81. google scholar
21. Ketola, A. (2016b). Translating picturebooks: Re-examining interlingual and intersemiotic translation. In Lloyd, P. and Bohemia, E. (eds.), Future Focused Thinking - DRS International Conference 2016, 27 - 30 June, Brighton, United Kingdom. https://doi.org/10.21606/ drs.2016.225 google scholar
22. Ketola, A. (2018). Word-image interaction in technical translation. Students translating an illustrated text. Unpublished PhD Dissertation, University of Tampere, Tampere google scholar
23. Kokkola, S., & Ketola, A. (2015). Thinking outside the “Methods Box”: New avenues for research in multimodal translation. In D. Rellstab google scholar
24. & N. Siponkoski (Eds.), Rajojen dynamiikkaa, Gränsernas dynamik, Borders under negotiation, Grenzen und ihre Dynamik. VAKKI-symposiumi XXXV 12.– 13.2.2015 (pp. 219–228). Vaasa: VAKKI Publications 4. google scholar
25. Kress, G. & Van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Arnold. google scholar
26. Kress, G. & Van Leeuwen, T. (2006). Reading Images: The Grammar of Visual Design (2nd ed.). Taylor & Francis Group, New York google scholar
27. Kress, G. (2003). Literacy in the new media age. Modes, logics and affordances. London: Routledge. google scholar
28. Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. Routledge. google scholar
29. Kress, G. (2020). Transposing meaning: translation in a multimodal semiotic landscape. In M. Boria, Á. Carreres, M. Noriega-Sánchez, & google scholar
30. M. Tomalin, (Eds), Translation and Multimodality. Beyond Words (pp. 24-48), Routledge: New York. google scholar
31. Li, S. (2019). A corpus-based multimodal approach to the translation of restaurant menus, Perspectives, 27(1), 1-19, DOI: 10.1080/0907676X.2018.1483408 google scholar
32. Li, Y., Shimizu, S., Chu, C., Kurohashi, S., & Li, W. (2023). Video-helpful multimodal machine translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 4281–4299), Singapore. Association for Computational Linguistics. google scholar
33. Liu, X., Zhao, J., Sun, S., Liu, H., & Yang, H. (2021). Variational multimodal machine translation with underlying semantic alignment. Information Fusion, 69, 73-80. https://doi.org/10.1016/j.inffus.2020.11.011 google scholar
34. Lu, S. (2023). How existing literary translation fits into film adaptations: the subtitling of neologisms in Harry Potter from a multimodal perspective. Visual Communication, 0(0), https://doi.org/10.1177/14703572221141959 google scholar
35. Mejías-Climent, L. (2017). Multimodality and Dubbing in Video Games: A Research Approach. Linguistica Antverpiensia, New Series: Themes in Translation Studies, 17, 99–113. google scholar
36. Nam, W. & Jang, B. (2024). A survey on multimodal bidirectional machine learning translation of image and natural language processing. google scholar
37. Expert Systems with Applications, 235(2024) 121168, 1-14. https://doi.org/10.1016/j.eswa.2023.121168 google scholar
38. O’Halloran, K. L., Tan, S. & Wignell, P. (2016). Intersemiotic translation as resemiotisation: A multimodal perspective. Signata, 7, 199-229. google scholar
39. Okyayuz, A. Ş., & Sancaktaroğlu Bozkurt, S. (2022). Çok modluluk ve çeviri: Nasreddin Hoca örneği. Avrasya Uluslararası Araştırmalar Dergisi, 10(32), 176-193. https://doi.org/10.33692/avrasyad.1176853 google scholar
40. Ouyang, Q. & Fu, A. (2021). Effects of non-verbal paralanguage capturing on meaning transfer in consecutive interpreting. In M. Zhang & D. W. Feng (Eds.), Multimodal approaches to Chinese-English translation and interpreting (pp. 192–218). Routledge, New York. google scholar
41. Painter, C., Martin, J. R., & Unsworth, L. (2013). Reading visual narratives. London: Equinox. google scholar
42. Pan, L. & Liao, S. (2023). Modelling audiovisual translation of non-fiction videos: A multimodal approach to subtitling. In L. Pan, X. Wu, T. Luo, & H. Qian (Eds), Multimodality in Translation Studies (pp. 21-34). Routledge: London and New York. google scholar
43. Pedro Ricoy, R. (2012). Multimodality in translation: Steps towards socially useful research. Journal Multimodal Communication, 1(2), 181-203. https://doi.org/10.1515/mc-2012-0012 google scholar
44. Pérez González, L. (2014). Multimodality in translation and interpreting studies: Theoretical and methodological perspectives. In S. Bermann & C. Porter (Eds.), A Companion to Translation Studies (pp. 119–131). Wiley. google scholar
45. Plastina, A. F. (2022). Remaking meaning through intersemiotic translation: the case of online medical journals. The Journal of Specialised Translation, 38, 32-54. google scholar
46. Povoroznyuk, R. (2022). Intersemiotics of multimodality: Advertisement in translation. Journal of Language and Linguistic Studies, 18(Special Issue 2), 1141-1164. google scholar
47. Ramos Pinto, S. (2018). Film, dialects and subtitles: An analytical framework for the study of non-standard varieties in subtitling. The Translator 24(1), 17-34 google scholar
48. Reviers, N. (2018). Tracking multimodal cohesion in audio description: Examples from a Dutch audio description corpus. Linguistica Antverpiensia, New Series – Themes in Translation Studies, 17, 22–35. https://doi.org/10.52034/lanstts.v17i0.477 google scholar
49. Sanabria, R., Çağlayan, O., Palaskar, S., Elliott, D., Barrault, L., Specia, L., & Metze, F. (2018). How2: A large-scale dataset for multimodal language understanding. In Proceedings of the Work-shop on Visually Grounded Interaction and Language (ViGIL). NeurIPS. google scholar
50. Silvester, H. (2018). From paratext to polysemiotic network: A holistic approach to the study of subtitled films. Linguistica Antverpiensia, New Series: Themes in Translation Studies, 14, 71–83. google scholar
51. Sulubacak, U., Çağlayan, O., Grönroos, S.-A., Rouhe, A., Elliott, D., Specia, L., & Tiedemann, J. (2020). Multimodal machine translation through visuals and speech. Machine Translation, 34, 97–147 (2020). https://doi.org/10.1007/s10590-020-09250-0 google scholar
52. Taylor, C. (2003). Multimodal transcription in the analysis, translation and subtitling of Italian films. The Translator, 9(2), 191–205. https:// doi.org/10.1080/13556509.2003.10799153 google scholar
53. Taylor, C. (2016). Multimodal corpus analysis of subtitling: The case of non-standard varieties. Target. International Journal of Translation Studies, 28(2), 222–236. https://doi.org/10.1075/target.28.2.04tay google scholar
54. Thibault, P. (2000). The multimodal transcription of a television advertisement: theory and practice. In A. Baldry (ed.) Multimodality and Multimediality in the Distance Learning Age (pp. 311–385). Campobasso: Palladino Editore. google scholar
55. Toury, G. (1994). A Cultural-Semiotic Perspective. In Thomas A. Sebeok (ed), Encyclopedic Dictionary of Semiotics, vol. 2 N-Z.(pp. 1111– 1124). Berlin and New York: Mouton de Gruyter. google scholar
56. Toury, G. (1995). Descriptive Translation Studies and beyond. Amsterdam and Philadelphia: John Benjamins. google scholar
57. Tuominen, T., Jiménez Hurtado, C., Ketola, A. (2018). Why methods matter: Approaching multimodality in translation research. Linguistica Antverpiensia, New Series: Themes in Translation Studies, 17, 1–21. google scholar
58. Valdeón, R. A. (2024). The translation of multimodal texts: challenges and theoretical approaches, Perspectives, 32(1), 1-13, DOI: 10.1080/0907676X.2024.2290928 google scholar
59. Van Leeuwen,T. (2005). Introducing Social Semiotics. London and New York: Routledge. google scholar
60. Van Meerbergen, S. (2009). Dutch picture books in Swedish translation. Towards a model for multimodal analysis. In D. C., Dries (ed.). Translation and (Trans)formation of Identities. Selected papers of the CETRA Research Seminar in Translation Studies, (pp. 1-20) Lirias: CETAR. google scholar
61. Wang, X., Wu, J., Chen, J., Li, L., Wang, Y., & Wang, W. Y. (2019). VATEX: a large-scale, high-quality multilingual dataset for video-and-language research. Computing research repository arXiv:1904.03493 google scholar
62. Weissbrod, R. & Kohn, A. (2019). Introduction. In R. Weissbrod, & A. Kohn (Eds), Translating the Visual. A Multimodal Perspective (pp. 1-12). New York, Routledge. google scholar
63. Wu, X., & Pan, L. (2023). Introduction: Multimodality in Translation Studies: Themes and Models. In L. Pan, X. Wu, T. Luo, & H. Qian (Eds), Multimodality in Translation Studies (pp. 1-20). London and New York: Routledge google scholar
64. Young, O., Lai, A., Hodosh, M. & Hockenmaier, J. (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2, 67-78. google scholar
65. Zhao, Y., Komachi, M., Kajiwara, T., & Chu, C. (2020). Double attention-based multimodal neural machine translation with semantic image regions. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pp. 105–114, Lisbon, Portugal. European Association for Machine Translation google scholar
66. Zhao, J. (2023). Re-instantiation of multimodal meaning in the translation of Chinese comic Zhuang Zi Speaks: The Music of Nature. Asia Pacific Translation and Intercultural Studies, 10(1), 34-52. google scholar