TY  - JOUR
T1  - Multimodal Change in Translation Studies: Transcending the Boundaries
TT  - Çeviribilimde Çok Modlu Değişim: Sınırları Aşmak
AU  - Balaman, Sevda
PY  - 2025
DA  - July
Y2  - 2024
DO  - 10.26650/iujts.2025.1551604
JF  - IU Journal of Translation Studies
PB  - Istanbul University
WT  - DergiPark
SN  - 2717-6959
SP  - 277
EP  - 290
IS  - 22
LA  - en
AB  - Upon recognizing the changing landscape of the communication register in the modern era with the burgeoning impacts of emerging technologies, the scope of translation research has been redefined in a way that embraces the prominence of non-verbal modes, alongside the linguistic code in meaning-construction. In this sense, the multimodality concept has increasingly received attention as an indispensable component of the meaning-transfer process, and its bidirectional connection with Translation Studies has been specially emphasized as a new research avenue to capture a clear image of the convergence in these fields. However, there exists a dearth of studies in the related literature, specifically devoted to the functionality of multimodal cohesion in translations. With this in mind, in light of previous research, this review study seeks to determine (a) how the phenomenon of multimodality locates itself in translation, (b) how it structures the extent of translatology research, (c) how multimodal texts are deciphered with reference to well-established transcribing methods, and (d) the current directions that the trans lation research in this paradigm is moving towards. For all these purposes, starting with an in-depth elaboration of conceptual issues by addressing the underlying core theories, in the methodology section of this paper, a number of theoretical frameworks used in different genres were presented to specify how a multimodal message is best transferred. Overall, this study can add new dimensions to this discipline by enabling translators to gain more awareness of the multimodal stratification of texts and the relevant translation methods.
KW  - audio-visual translation
KW  - mode
KW  - multimodality
KW  - non-verbal modalities
KW  - social semiotics
N2  - Gelişen teknolojilerin artan etkileri ile bağlantılı olarak, modern çağda iletişim dilinin değişen yapısının farkına varılması üzerine çeviri araştırmasının kapsamı, anlamı oluşturmada dilbilimsel modun yanı sıra sözel olmayan biçimlerin de önemini benimseyecek bir şekilde yeniden tanımlanmıştır. Bu anlamda, çok modluluk kavramı, anlam aktarım sürecinin vazgeçilmez bir bileşeni olarak, gün geçtikçe daha da dikkat çekmiştir ve bu kavram ile Çeviribilim arasında bulunan çift yönlü bağlantı, söz konusu alanlardaki yakınsamanın net bir görüntüsünü yakalamak için yeni bir araştırma alanı olarak özellikle vurgulanmıştır. Bununla birlikte, ilgili alan yazında, özellikle çevirilerde çok modlu bütünlüğün işlevselliğine yönelik çalışmaların eksikliği söz konusudur. Bu durumu akılda tutarak, bu inceleme çalışması, önceki araştırmaların ışığında, (a) çok modluluk olgusunun çeviride kendisini nasıl konumlandırdığını, (b) çeviri araştırmasının kapsama alanını nasıl yapılandırdığını, (c) çok modlu metinlerin köklü çözümleme yöntemleri referans alınarak nasıl deşifre edildiğini ve (d) bu paradigmadaki çeviri araştırmasının ilerlediği mevcut yönleri belirlemeyi hedeflemektedir. Tüm bu amaçlar doğrultusunda, temel teşkil eden başlıca teorilere değinilerek, kavramsal konuların derinlemesine detaylandırılmasından başlayarak, bu makalenin yöntem bölümünde, çok modlu bir mesajın en iyi şekilde nasıl aktarılacağını belirlemek için farklı türlerde kullanılan bir takım teorik çerçeveler sunulmuştur. Genel olarak bu çalışma, çevirmenlerin metinlerin çok modlu katmanlaşması ve ilgili çeviri yöntemleri hakkında daha fazla farkındalık kazanmalarını sağlayarak bu alana yeni boyutlar kazandırabilir.
CR  - Adami, E. & Ramos Pinto, S. (2020). Meaning-(re)making in a world of untranslated signs: towards a research agenda on multimodality, culture, and translation. In M. Boria, Á. Carreres, M. Noriega Sánchez, & M. Tomalin (Eds.), Translation and multimodality: Beyond words (pp. 71–93). Routledge: New York. google scholar
CR  - Akiba, Y., Federico, M., Kando, N., Nakaiwa, H., Paul, M., & Tsujii, J. (2004). Overview of the IWSLT 2004 evaluation campaign. In Proceedings of the 2004 international workshop on spoken language translation (IWSLT), Kyoto, pp. 1–12. google scholar
CR  - Boria, M., & Tomalin, M. (2020). Introduction. In M. Boria, Á. Carreres, M. Noriega Sánchez, & M. Tomalin (Eds.), Translation and multimodality: Beyond words (pp. 1–23). Routledge: New York. google scholar
CR  - Carreres, Á. & Noriega-Sánchez, M. (2020). Beyond words: concluding remarks. In M. Boria, Á. Carreres, M. Noriega-Sánchez, & M. Tomalin, (Eds), Translation and Multimodality. Beyond Words (pp. 198-203), Routledge: New York. google scholar
CR  - Chen, X. (2023). Representing cityscape through texts and images: translations of multimodal public notices in Macao, Asia Pacific Translation and Intercultural Studies, 10(1), 53-70. google scholar
CR  - Chen, Y. & Wang, W. (2016). Relating visual images to subtitle translation in Finding Nemo: A multi-semiotic interplay. Translation & Interpreting, 8(1), 69-85. google scholar
CR  - Çağlayan, O. (2019). Multimodal machine translation. Computation and Language (Unpublished PhD Dissertation). Le Mans Université, France. google scholar
CR  - Damaskinidis, G. (2016). The visual aspect of translation training in multimodal texts. Meta, 61(2), 299–319. https://doi.org/10.7202/ 1037761ar google scholar
CR  - Elliott, D., Frank, S., Sima’an, K., & Specia. L. (2016). Multi30K: Multilingual English- German image descriptions. In Proceedings of the 5th Workshop on Vision and Language (pp. 70– 74), Berlin, Germany. Association for Computational Linguistics. google scholar
CR  - Everett, Y. U. (2015). Reconfiguring myth and narrative in Contemporary opera. Bloomington: Indiana University Press. google scholar
CR  - Gambier, Y. (2006). Multimodality and Audiovisual Translation. Proceedings of the Marie Curie Euroconferences MuTra: Audiovisual translation scenarios (pp. 1-8). Retrieved from http://euroconferences.info/proceedings/2006_Proceedings/2006_Gambier_Yves. pdf google scholar
CR  - Gambier, Y. (2023). Audiovisual translation and multimodality: What future? Media and Intercultural Communication: A Multidisciplinary Journal, 1(1), 1-16. https://doi.org/10.22034/mic.2023.167451 google scholar
CR  - Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Arnold. google scholar
CR  - Hodge, R. & Kress, G. (1988). Social semiotics. Ithaca, NY: Cornell University Press. google scholar
CR  - Jakobson, R. (1959). On linguistic aspects of translation. In L. Venuti (Ed.), The Translation Studies Reader (pp. 232–239). London: Routledge. google scholar
CR  - Jewitt, C., Bezemer, J., & O’Halloran, K. L. (2016). Introducing multimodality. London: Routledge. google scholar
CR  - Kaindl, K. (2013). Multimodality and translation. In C. Millán & F. Bartrina (Eds.), The Routledge Handbook of Translation Studies (pp. 257–269). London: Routledge. google scholar
CR  - Kaindl, K. (2020). A theoretical framework for a multimodal conception of translation. In M. Boria, Á. Carreres, M. Noriega-Sánchez, & M. google scholar
CR  - Tomalin, (Eds), Translation and Multimodality. Beyond Words (pp. 49-70), Routledge: New York. google scholar
CR  - Ketola, A. (2016a). Towards a multimodally oriented theory of translation: A cognitive framework for the translation of illustrated technical texts. Translation Studies, 9(1), 67–81. google scholar
CR  - Ketola, A. (2016b). Translating picturebooks: Re-examining interlingual and intersemiotic translation. In Lloyd, P. and Bohemia, E. (eds.), Future Focused Thinking - DRS International Conference 2016, 27 - 30 June, Brighton, United Kingdom. https://doi.org/10.21606/ drs.2016.225 google scholar
CR  - Ketola, A. (2018). Word-image interaction in technical translation. Students translating an illustrated text. Unpublished PhD Dissertation, University of Tampere, Tampere google scholar
CR  - Kokkola, S., & Ketola, A. (2015). Thinking outside the “Methods Box”: New avenues for research in multimodal translation. In D. Rellstab google scholar
CR  - & N. Siponkoski (Eds.), Rajojen dynamiikkaa, Gränsernas dynamik, Borders under negotiation, Grenzen und ihre Dynamik. VAKKI-symposiumi XXXV 12.– 13.2.2015 (pp. 219–228). Vaasa: VAKKI Publications 4. google scholar
CR  - Kress, G. & Van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Arnold. google scholar
CR  - Kress, G. & Van Leeuwen, T. (2006). Reading Images: The Grammar of Visual Design (2nd ed.). Taylor & Francis Group, New York google scholar
CR  - Kress, G. (2003). Literacy in the new media age. Modes, logics and affordances. London: Routledge. google scholar
CR  - Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. Routledge. google scholar
CR  - Kress, G. (2020). Transposing meaning: translation in a multimodal semiotic landscape. In M. Boria, Á. Carreres, M. Noriega-Sánchez, & google scholar
CR  - M. Tomalin, (Eds), Translation and Multimodality. Beyond Words (pp. 24-48), Routledge: New York. google scholar
CR  - Li, S. (2019). A corpus-based multimodal approach to the translation of restaurant menus, Perspectives, 27(1), 1-19, DOI: 10.1080/0907676X.2018.1483408 google scholar
CR  - Li, Y., Shimizu, S., Chu, C., Kurohashi, S., & Li, W. (2023). Video-helpful multimodal machine translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 4281–4299), Singapore. Association for Computational Linguistics. google scholar
CR  - Liu, X., Zhao, J., Sun, S., Liu, H., & Yang, H. (2021). Variational multimodal machine translation with underlying semantic alignment. Information Fusion, 69, 73-80. https://doi.org/10.1016/j.inffus.2020.11.011 google scholar
CR  - Lu, S. (2023). How existing literary translation fits into film adaptations: the subtitling of neologisms in Harry Potter from a multimodal perspective. Visual Communication, 0(0), https://doi.org/10.1177/14703572221141959 google scholar
CR  - Mejías-Climent, L. (2017). Multimodality and Dubbing in Video Games: A Research Approach. Linguistica Antverpiensia, New Series: Themes in Translation Studies, 17, 99–113. google scholar
CR  - Nam, W. & Jang, B. (2024). A survey on multimodal bidirectional machine learning translation of image and natural language processing. google scholar
CR  - Expert Systems with Applications, 235(2024) 121168, 1-14. https://doi.org/10.1016/j.eswa.2023.121168 google scholar
CR  - O’Halloran, K. L., Tan, S. & Wignell, P. (2016). Intersemiotic translation as resemiotisation: A multimodal perspective. Signata, 7, 199-229. google scholar
CR  - Okyayuz, A. Ş., & Sancaktaroğlu Bozkurt, S. (2022). Çok modluluk ve çeviri: Nasreddin Hoca örneği. Avrasya Uluslararası Araştırmalar Dergisi, 10(32), 176-193. https://doi.org/10.33692/avrasyad.1176853 google scholar
CR  - Ouyang, Q. & Fu, A. (2021). Effects of non-verbal paralanguage capturing on meaning transfer in consecutive interpreting. In M. Zhang & D. W. Feng (Eds.), Multimodal approaches to Chinese-English translation and interpreting (pp. 192–218). Routledge, New York. google scholar
CR  - Painter, C., Martin, J. R., & Unsworth, L. (2013). Reading visual narratives. London: Equinox. google scholar
CR  - Pan, L. & Liao, S. (2023). Modelling audiovisual translation of non-fiction videos: A multimodal approach to subtitling. In L. Pan, X. Wu, T. Luo, & H. Qian (Eds), Multimodality in Translation Studies (pp. 21-34). Routledge: London and New York. google scholar
CR  - Pedro Ricoy, R. (2012). Multimodality in translation: Steps towards socially useful research. Journal Multimodal Communication, 1(2), 181-203. https://doi.org/10.1515/mc-2012-0012 google scholar
CR  - Pérez González, L. (2014). Multimodality in translation and interpreting studies: Theoretical and methodological perspectives. In S. Bermann & C. Porter (Eds.), A Companion to Translation Studies (pp. 119–131). Wiley. google scholar
CR  - Plastina, A. F. (2022). Remaking meaning through intersemiotic translation: the case of online medical journals. The Journal of Specialised Translation, 38, 32-54. google scholar
CR  - Povoroznyuk, R. (2022). Intersemiotics of multimodality: Advertisement in translation. Journal of Language and Linguistic Studies, 18(Special Issue 2), 1141-1164. google scholar
CR  - Ramos Pinto, S. (2018). Film, dialects and subtitles: An analytical framework for the study of non-standard varieties in subtitling. The Translator 24(1), 17-34 google scholar
CR  - Reviers, N. (2018). Tracking multimodal cohesion in audio description: Examples from a Dutch audio description corpus. Linguistica Antverpiensia, New Series – Themes in Translation Studies, 17, 22–35. https://doi.org/10.52034/lanstts.v17i0.477 google scholar
CR  - Sanabria, R., Çağlayan, O., Palaskar, S., Elliott, D., Barrault, L., Specia, L., & Metze, F. (2018). How2: A large-scale dataset for multimodal language understanding. In Proceedings of the Work-shop on Visually Grounded Interaction and Language (ViGIL). NeurIPS. google scholar
CR  - Silvester, H. (2018). From paratext to polysemiotic network: A holistic approach to the study of subtitled films. Linguistica Antverpiensia, New Series: Themes in Translation Studies, 14, 71–83. google scholar
CR  - Sulubacak, U., Çağlayan, O., Grönroos, S.-A., Rouhe, A., Elliott, D., Specia, L., & Tiedemann, J. (2020). Multimodal machine translation through visuals and speech. Machine Translation, 34, 97–147 (2020). https://doi.org/10.1007/s10590-020-09250-0 google scholar
CR  - Taylor, C. (2003). Multimodal transcription in the analysis, translation and subtitling of Italian films. The Translator, 9(2), 191–205. https:// doi.org/10.1080/13556509.2003.10799153 google scholar
CR  - Taylor, C. (2016). Multimodal corpus analysis of subtitling: The case of non-standard varieties. Target. International Journal of Translation Studies, 28(2), 222–236. https://doi.org/10.1075/target.28.2.04tay google scholar
CR  - Thibault, P. (2000). The multimodal transcription of a television advertisement: theory and practice. In A. Baldry (ed.) Multimodality and Multimediality in the Distance Learning Age (pp. 311–385). Campobasso: Palladino Editore. google scholar
CR  - Toury, G. (1994). A Cultural-Semiotic Perspective. In Thomas A. Sebeok (ed), Encyclopedic Dictionary of Semiotics, vol. 2 N-Z.(pp. 1111– 1124). Berlin and New York: Mouton de Gruyter. google scholar
CR  - Toury, G. (1995). Descriptive Translation Studies and beyond. Amsterdam and Philadelphia: John Benjamins. google scholar
CR  - Tuominen, T., Jiménez Hurtado, C., Ketola, A. (2018). Why methods matter: Approaching multimodality in translation research. Linguistica Antverpiensia, New Series: Themes in Translation Studies, 17, 1–21. google scholar
CR  - Valdeón, R. A. (2024). The translation of multimodal texts: challenges and theoretical approaches, Perspectives, 32(1), 1-13, DOI: 10.1080/0907676X.2024.2290928 google scholar
CR  - Van Leeuwen,T. (2005). Introducing Social Semiotics. London and New York: Routledge. google scholar
CR  - Van Meerbergen, S. (2009). Dutch picture books in Swedish translation. Towards a model for multimodal analysis. In D. C., Dries (ed.). Translation and (Trans)formation of Identities. Selected papers of the CETRA Research Seminar in Translation Studies, (pp. 1-20) Lirias: CETAR. google scholar
CR  - Wang, X., Wu, J., Chen, J., Li, L., Wang, Y., & Wang, W. Y. (2019). VATEX: a large-scale, high-quality multilingual dataset for video-and-language research. Computing research repository arXiv:1904.03493 google scholar
CR  - Weissbrod, R. & Kohn, A. (2019). Introduction. In R. Weissbrod, & A. Kohn (Eds), Translating the Visual. A Multimodal Perspective (pp. 1-12). New York, Routledge. google scholar
CR  - Wu, X., & Pan, L. (2023). Introduction: Multimodality in Translation Studies: Themes and Models. In L. Pan, X. Wu, T. Luo, & H. Qian (Eds), Multimodality in Translation Studies (pp. 1-20). London and New York: Routledge google scholar
CR  - Young, O., Lai, A., Hodosh, M. & Hockenmaier, J. (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2, 67-78. google scholar
CR  - Zhao, Y., Komachi, M., Kajiwara, T., & Chu, C. (2020). Double attention-based multimodal neural machine translation with semantic image regions. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pp. 105–114, Lisbon, Portugal. European Association for Machine Translation google scholar
CR  - Zhao, J. (2023). Re-instantiation of multimodal meaning in the translation of Chinese comic Zhuang Zi Speaks: The Music of Nature. Asia Pacific Translation and Intercultural Studies, 10(1), 34-52. google scholar
UR  - https://doi.org/10.26650/iujts.2025.1551604
L1  - https://dergipark.org.tr/en/download/article-file/4220977
ER  -