Çeviribilimde Çok Modlu Değişim: Sınırları Aşmak

Year 2025, Issue: 22, 277 - 290, 04.07.2025

Sevda Balaman

https://doi.org/10.26650/iujts.2025.1551604

Abstract

Gelişen teknolojilerin artan etkileri ile bağlantılı olarak, modern çağda iletişim dilinin değişen yapısının farkına varılması üzerine çeviri araştırmasının kapsamı, anlamı oluşturmada dilbilimsel modun yanı sıra sözel olmayan biçimlerin de önemini benimseyecek bir şekilde yeniden tanımlanmıştır. Bu anlamda, çok modluluk kavramı, anlam aktarım sürecinin vazgeçilmez bir bileşeni olarak, gün geçtikçe daha da dikkat çekmiştir ve bu kavram ile Çeviribilim arasında bulunan çift yönlü bağlantı, söz konusu alanlardaki yakınsamanın net bir görüntüsünü yakalamak için yeni bir araştırma alanı olarak özellikle vurgulanmıştır. Bununla birlikte, ilgili alan yazında, özellikle çevirilerde çok modlu bütünlüğün işlevselliğine yönelik çalışmaların eksikliği söz konusudur. Bu durumu akılda tutarak, bu inceleme çalışması, önceki araştırmaların ışığında, (a) çok modluluk olgusunun çeviride kendisini nasıl konumlandırdığını, (b) çeviri araştırmasının kapsama alanını nasıl yapılandırdığını, (c) çok modlu metinlerin köklü çözümleme yöntemleri referans alınarak nasıl deşifre edildiğini ve (d) bu paradigmadaki çeviri araştırmasının ilerlediği mevcut yönleri belirlemeyi hedeflemektedir. Tüm bu amaçlar doğrultusunda, temel teşkil eden başlıca teorilere değinilerek, kavramsal konuların derinlemesine detaylandırılmasından başlayarak, bu makalenin yöntem bölümünde, çok modlu bir mesajın en iyi şekilde nasıl aktarılacağını belirlemek için farklı türlerde kullanılan bir takım teorik çerçeveler sunulmuştur. Genel olarak bu çalışma, çevirmenlerin metinlerin çok modlu katmanlaşması ve ilgili çeviri yöntemleri hakkında daha fazla farkındalık kazanmalarını sağlayarak bu alana yeni boyutlar kazandırabilir.

Keywords

çok modluluk , görsel-işitsel çeviri , mod , sosyal göstergebilim , sözlü olmayan modlar

Multimodal Change in Translation Studies: Transcending the Boundaries

Abstract

Upon recognizing the changing landscape of the communication register in the modern era with the burgeoning impacts of emerging technologies, the scope of translation research has been redefined in a way that embraces the prominence of non-verbal modes, alongside the linguistic code in meaning-construction. In this sense, the multimodality concept has increasingly received attention as an indispensable component of the meaning-transfer process, and its bidirectional connection with Translation Studies has been specially emphasized as a new research avenue to capture a clear image of the convergence in these fields. However, there exists a dearth of studies in the related literature, specifically devoted to the functionality of multimodal cohesion in translations. With this in mind, in light of previous research, this review study seeks to determine (a) how the phenomenon of multimodality locates itself in translation, (b) how it structures the extent of translatology research, (c) how multimodal texts are deciphered with reference to well-established transcribing methods, and (d) the current directions that the trans lation research in this paradigm is moving towards. For all these purposes, starting with an in-depth elaboration of conceptual issues by addressing the underlying core theories, in the methodology section of this paper, a number of theoretical frameworks used in different genres were presented to specify how a multimodal message is best transferred. Overall, this study can add new dimensions to this discipline by enabling translators to gain more awareness of the multimodal stratification of texts and the relevant translation methods.

Keywords

audio-visual translation , mode , multimodality , non-verbal modalities , social semiotics

References

• Adami, E. & Ramos Pinto, S. (2020). Meaning-(re)making in a world of untranslated signs: towards a research agenda on multimodality, culture, and translation. In M. Boria, Á. Carreres, M. Noriega Sánchez, & M. Tomalin (Eds.), Translation and multimodality: Beyond words (pp. 71–93). Routledge: New York. google scholar
• Akiba, Y., Federico, M., Kando, N., Nakaiwa, H., Paul, M., & Tsujii, J. (2004). Overview of the IWSLT 2004 evaluation campaign. In Proceedings of the 2004 international workshop on spoken language translation (IWSLT), Kyoto, pp. 1–12. google scholar
• Boria, M., & Tomalin, M. (2020). Introduction. In M. Boria, Á. Carreres, M. Noriega Sánchez, & M. Tomalin (Eds.), Translation and multimodality: Beyond words (pp. 1–23). Routledge: New York. google scholar
• Carreres, Á. & Noriega-Sánchez, M. (2020). Beyond words: concluding remarks. In M. Boria, Á. Carreres, M. Noriega-Sánchez, & M. Tomalin, (Eds), Translation and Multimodality. Beyond Words (pp. 198-203), Routledge: New York. google scholar
• Chen, X. (2023). Representing cityscape through texts and images: translations of multimodal public notices in Macao, Asia Pacific Translation and Intercultural Studies, 10(1), 53-70. google scholar
• Chen, Y. & Wang, W. (2016). Relating visual images to subtitle translation in Finding Nemo: A multi-semiotic interplay. Translation & Interpreting, 8(1), 69-85. google scholar
• Çağlayan, O. (2019). Multimodal machine translation. Computation and Language (Unpublished PhD Dissertation). Le Mans Université, France. google scholar
• Damaskinidis, G. (2016). The visual aspect of translation training in multimodal texts. Meta, 61(2), 299–319. https://doi.org/10.7202/ 1037761ar google scholar
• Elliott, D., Frank, S., Sima’an, K., & Specia. L. (2016). Multi30K: Multilingual English- German image descriptions. In Proceedings of the 5th Workshop on Vision and Language (pp. 70– 74), Berlin, Germany. Association for Computational Linguistics. google scholar
• Everett, Y. U. (2015). Reconfiguring myth and narrative in Contemporary opera. Bloomington: Indiana University Press. google scholar
• Gambier, Y. (2006). Multimodality and Audiovisual Translation. Proceedings of the Marie Curie Euroconferences MuTra: Audiovisual translation scenarios (pp. 1-8). Retrieved from http://euroconferences.info/proceedings/2006_Proceedings/2006_Gambier_Yves. pdf google scholar
• Gambier, Y. (2023). Audiovisual translation and multimodality: What future? Media and Intercultural Communication: A Multidisciplinary Journal, 1(1), 1-16. https://doi.org/10.22034/mic.2023.167451 google scholar
• Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Arnold. google scholar
• Hodge, R. & Kress, G. (1988). Social semiotics. Ithaca, NY: Cornell University Press. google scholar
• Jakobson, R. (1959). On linguistic aspects of translation. In L. Venuti (Ed.), The Translation Studies Reader (pp. 232–239). London: Routledge. google scholar
• Jewitt, C., Bezemer, J., & O’Halloran, K. L. (2016). Introducing multimodality. London: Routledge. google scholar
• Kaindl, K. (2013). Multimodality and translation. In C. Millán & F. Bartrina (Eds.), The Routledge Handbook of Translation Studies (pp. 257–269). London: Routledge. google scholar
• Kaindl, K. (2020). A theoretical framework for a multimodal conception of translation. In M. Boria, Á. Carreres, M. Noriega-Sánchez, & M. google scholar
• Tomalin, (Eds), Translation and Multimodality. Beyond Words (pp. 49-70), Routledge: New York. google scholar
• Ketola, A. (2016a). Towards a multimodally oriented theory of translation: A cognitive framework for the translation of illustrated technical texts. Translation Studies, 9(1), 67–81. google scholar
• Ketola, A. (2016b). Translating picturebooks: Re-examining interlingual and intersemiotic translation. In Lloyd, P. and Bohemia, E. (eds.), Future Focused Thinking - DRS International Conference 2016, 27 - 30 June, Brighton, United Kingdom. https://doi.org/10.21606/ drs.2016.225 google scholar
• Ketola, A. (2018). Word-image interaction in technical translation. Students translating an illustrated text. Unpublished PhD Dissertation, University of Tampere, Tampere google scholar
• Kokkola, S., & Ketola, A. (2015). Thinking outside the “Methods Box”: New avenues for research in multimodal translation. In D. Rellstab google scholar
• & N. Siponkoski (Eds.), Rajojen dynamiikkaa, Gränsernas dynamik, Borders under negotiation, Grenzen und ihre Dynamik. VAKKI-symposiumi XXXV 12.– 13.2.2015 (pp. 219–228). Vaasa: VAKKI Publications 4. google scholar
• Kress, G. & Van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Arnold. google scholar
• Kress, G. & Van Leeuwen, T. (2006). Reading Images: The Grammar of Visual Design (2nd ed.). Taylor & Francis Group, New York google scholar
• Kress, G. (2003). Literacy in the new media age. Modes, logics and affordances. London: Routledge. google scholar
• Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. Routledge. google scholar
• Kress, G. (2020). Transposing meaning: translation in a multimodal semiotic landscape. In M. Boria, Á. Carreres, M. Noriega-Sánchez, & google scholar
• M. Tomalin, (Eds), Translation and Multimodality. Beyond Words (pp. 24-48), Routledge: New York. google scholar
• Li, S. (2019). A corpus-based multimodal approach to the translation of restaurant menus, Perspectives, 27(1), 1-19, DOI: 10.1080/0907676X.2018.1483408 google scholar
• Li, Y., Shimizu, S., Chu, C., Kurohashi, S., & Li, W. (2023). Video-helpful multimodal machine translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 4281–4299), Singapore. Association for Computational Linguistics. google scholar
• Liu, X., Zhao, J., Sun, S., Liu, H., & Yang, H. (2021). Variational multimodal machine translation with underlying semantic alignment. Information Fusion, 69, 73-80. https://doi.org/10.1016/j.inffus.2020.11.011 google scholar
• Lu, S. (2023). How existing literary translation fits into film adaptations: the subtitling of neologisms in Harry Potter from a multimodal perspective. Visual Communication, 0(0), https://doi.org/10.1177/14703572221141959 google scholar
• Mejías-Climent, L. (2017). Multimodality and Dubbing in Video Games: A Research Approach. Linguistica Antverpiensia, New Series: Themes in Translation Studies, 17, 99–113. google scholar
• Nam, W. & Jang, B. (2024). A survey on multimodal bidirectional machine learning translation of image and natural language processing. google scholar
• Expert Systems with Applications, 235(2024) 121168, 1-14. https://doi.org/10.1016/j.eswa.2023.121168 google scholar
• O’Halloran, K. L., Tan, S. & Wignell, P. (2016). Intersemiotic translation as resemiotisation: A multimodal perspective. Signata, 7, 199-229. google scholar
• Okyayuz, A. Ş., & Sancaktaroğlu Bozkurt, S. (2022). Çok modluluk ve çeviri: Nasreddin Hoca örneği. Avrasya Uluslararası Araştırmalar Dergisi, 10(32), 176-193. https://doi.org/10.33692/avrasyad.1176853 google scholar
• Ouyang, Q. & Fu, A. (2021). Effects of non-verbal paralanguage capturing on meaning transfer in consecutive interpreting. In M. Zhang & D. W. Feng (Eds.), Multimodal approaches to Chinese-English translation and interpreting (pp. 192–218). Routledge, New York. google scholar
• Painter, C., Martin, J. R., & Unsworth, L. (2013). Reading visual narratives. London: Equinox. google scholar
• Pan, L. & Liao, S. (2023). Modelling audiovisual translation of non-fiction videos: A multimodal approach to subtitling. In L. Pan, X. Wu, T. Luo, & H. Qian (Eds), Multimodality in Translation Studies (pp. 21-34). Routledge: London and New York. google scholar
• Pedro Ricoy, R. (2012). Multimodality in translation: Steps towards socially useful research. Journal Multimodal Communication, 1(2), 181-203. https://doi.org/10.1515/mc-2012-0012 google scholar
• Pérez González, L. (2014). Multimodality in translation and interpreting studies: Theoretical and methodological perspectives. In S. Bermann & C. Porter (Eds.), A Companion to Translation Studies (pp. 119–131). Wiley. google scholar
• Plastina, A. F. (2022). Remaking meaning through intersemiotic translation: the case of online medical journals. The Journal of Specialised Translation, 38, 32-54. google scholar
• Povoroznyuk, R. (2022). Intersemiotics of multimodality: Advertisement in translation. Journal of Language and Linguistic Studies, 18(Special Issue 2), 1141-1164. google scholar
• Ramos Pinto, S. (2018). Film, dialects and subtitles: An analytical framework for the study of non-standard varieties in subtitling. The Translator 24(1), 17-34 google scholar
• Reviers, N. (2018). Tracking multimodal cohesion in audio description: Examples from a Dutch audio description corpus. Linguistica Antverpiensia, New Series – Themes in Translation Studies, 17, 22–35. https://doi.org/10.52034/lanstts.v17i0.477 google scholar
• Sanabria, R., Çağlayan, O., Palaskar, S., Elliott, D., Barrault, L., Specia, L., & Metze, F. (2018). How2: A large-scale dataset for multimodal language understanding. In Proceedings of the Work-shop on Visually Grounded Interaction and Language (ViGIL). NeurIPS. google scholar
• Silvester, H. (2018). From paratext to polysemiotic network: A holistic approach to the study of subtitled films. Linguistica Antverpiensia, New Series: Themes in Translation Studies, 14, 71–83. google scholar
• Sulubacak, U., Çağlayan, O., Grönroos, S.-A., Rouhe, A., Elliott, D., Specia, L., & Tiedemann, J. (2020). Multimodal machine translation through visuals and speech. Machine Translation, 34, 97–147 (2020). https://doi.org/10.1007/s10590-020-09250-0 google scholar
• Taylor, C. (2003). Multimodal transcription in the analysis, translation and subtitling of Italian films. The Translator, 9(2), 191–205. https:// doi.org/10.1080/13556509.2003.10799153 google scholar
• Taylor, C. (2016). Multimodal corpus analysis of subtitling: The case of non-standard varieties. Target. International Journal of Translation Studies, 28(2), 222–236. https://doi.org/10.1075/target.28.2.04tay google scholar
• Thibault, P. (2000). The multimodal transcription of a television advertisement: theory and practice. In A. Baldry (ed.) Multimodality and Multimediality in the Distance Learning Age (pp. 311–385). Campobasso: Palladino Editore. google scholar
• Toury, G. (1994). A Cultural-Semiotic Perspective. In Thomas A. Sebeok (ed), Encyclopedic Dictionary of Semiotics, vol. 2 N-Z.(pp. 1111– 1124). Berlin and New York: Mouton de Gruyter. google scholar
• Toury, G. (1995). Descriptive Translation Studies and beyond. Amsterdam and Philadelphia: John Benjamins. google scholar
• Tuominen, T., Jiménez Hurtado, C., Ketola, A. (2018). Why methods matter: Approaching multimodality in translation research. Linguistica Antverpiensia, New Series: Themes in Translation Studies, 17, 1–21. google scholar
• Valdeón, R. A. (2024). The translation of multimodal texts: challenges and theoretical approaches, Perspectives, 32(1), 1-13, DOI: 10.1080/0907676X.2024.2290928 google scholar
• Van Leeuwen,T. (2005). Introducing Social Semiotics. London and New York: Routledge. google scholar
• Van Meerbergen, S. (2009). Dutch picture books in Swedish translation. Towards a model for multimodal analysis. In D. C., Dries (ed.). Translation and (Trans)formation of Identities. Selected papers of the CETRA Research Seminar in Translation Studies, (pp. 1-20) Lirias: CETAR. google scholar
• Wang, X., Wu, J., Chen, J., Li, L., Wang, Y., & Wang, W. Y. (2019). VATEX: a large-scale, high-quality multilingual dataset for video-and-language research. Computing research repository arXiv:1904.03493 google scholar
• Weissbrod, R. & Kohn, A. (2019). Introduction. In R. Weissbrod, & A. Kohn (Eds), Translating the Visual. A Multimodal Perspective (pp. 1-12). New York, Routledge. google scholar
• Wu, X., & Pan, L. (2023). Introduction: Multimodality in Translation Studies: Themes and Models. In L. Pan, X. Wu, T. Luo, & H. Qian (Eds), Multimodality in Translation Studies (pp. 1-20). London and New York: Routledge google scholar
• Young, O., Lai, A., Hodosh, M. & Hockenmaier, J. (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2, 67-78. google scholar
• Zhao, Y., Komachi, M., Kajiwara, T., & Chu, C. (2020). Double attention-based multimodal neural machine translation with semantic image regions. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pp. 105–114, Lisbon, Portugal. European Association for Machine Translation google scholar
• Zhao, J. (2023). Re-instantiation of multimodal meaning in the translation of Chinese comic Zhuang Zi Speaks: The Music of Nature. Asia Pacific Translation and Intercultural Studies, 10(1), 34-52. google scholar