Assessing Quality in Machine-Translated English-Turkish Climate Communication Subtitles: A Longitudinal Study

Öz

As climate change communication increasingly relies on digital media, the quality of machine-translated subtitles becomes a critical factor in preventing misinformation. This study investigates the quality of English–Turkish machine-translated subtitles in climate change communication, a domain characterized by technical terminology and the need for high semantic precision. Focusing on two Intergovernmental Panel on Climate Change (IPCC) videos released in 2013 and 2022, we conduct a longitudinal analysis comparing their automatic subtitles from 2022 and 2025. Using Jan Pedersen’s (2017) FAR model—which assesses functional equivalence (semantic and stylistic accuracy), acceptability (grammatical and idiomatic correctness), and readability (technical presentation)—we offer a systematic assessment of translation quality and analyze how advances in machine translation (MT) have affected subtitle accuracy, fluency, and usability. Results demonstrate substantial improvement, with total error scores reduced by 63.04% for the first video and 53.40% for the second. These gains were uneven: the largest reduction in errors was in readability for the first video (73.77%) and in acceptability for the second (76.25%), while functional equivalence showed more limited progress (27.91%-53.18%) and remained the leading source of errors in 2025, with persistent issues in conveying nuanced meaning. The findings have implications for automated subtitle practices, translator training, and sustainability-focused media strategies. We argue that for producers of climate communication, a shift towards “a human-centered augmented translation” (O’Brien, 2023) model is necessary to reduce the risk of misinformation and improve the effectiveness of climate communication. Furthermore, for consumers, we contend that translation literacy, specifically MT literacy, is not merely a technical skill but a fundamental public competency, crucial for ensuring accessible and trustworthy communication. We recommend fostering this skill through educational initiatives beyond translation programs such as targeted workshops, short modules in relevant courses, and cross-departmental certificate options. This, in turn, will facilitate the capacity of non-specialist audiences to critically assess and comprehend MT-mediated digital content broadly, and information pertaining to climate conveyed via this modality specifically. The study highlights the role of high-quality translation in strengthening global climate literacy and supporting informed policy engagement by linking the evaluation of MT with environmental communication.

Anahtar Kelimeler

• machine translation
• audiovisual translation
• quality assessment
• machine translation literacy
• sustainability

Makine Çevirisiyle Üretilen İngilizce-Türkçe İklim İletişimi Altyazılarında Kalite Değerlendirmesi: Boylamsal Bir Araştırma

Öz

İklim değişikliği alanındaki iletişim giderek dijital medyaya daha fazla bağlı hâle gelirken, makine çevirisi (MÇ) altyazılarının kalitesi yanlış bilgilendirmenin önlenmesinde kritik rol oynamaktadır. Bu çalışma, teknik terminoloji açısından yoğun olan ve anlamsal açıdan yüksek hassasiyet gerektiren bu alanda İngilizce–Türkçe MÇ altyazılarının kalitesini incelemektedir. Bu kapsamda Hükümetlerarası İklim Değişikliği Paneli’nin (IPCC) 2013 ve 2022 yıllarında yayınlanan iki videosunun 2022 ve 2025 yıllarına ait otomatik altyazıları karşılaştırmalı ve boylamsal olarak analiz edilmiştir. Jan Pedersen’in (2017) işlevsel eşdeğerliği (anlamsal ve biçemsel doğruluk), kabul edilebilirliği (dilbilgisi, yazım ve dilin ifade özellikleri açısından uygunluk) ve okunabilirliği (teknik özellikler) ölçen FAR modeli kullanılarak altyazı kalitesi sistematik biçimde değerlendirilmiş; MÇ’deki gelişmelerin altyazı doğruluğu, akıcılığı ve kullanılabilirliği üzerindeki etkisi incelenmiştir. Bulgular, altyazı kalitesinde belirgin bir iyileşme olduğunu göstermektedir: ilk videoda toplam hata oranı %63,04, ikinci videoda ise %53,40 oranında azalmıştır. Bu iyileşmeler eşit dağılmamıştır; birinci videoda en yüksek azalma okunurlukta (%73,77), ikinci videoda ise kabul edilebilirlikte (%76,25) gözlemlenmiştir. İşlevsel eşdeğerlik ise daha sınırlı bir ilerleme göstermiş (birinci videoda %53,18, ikinci videoda %27,91) ve 2025 altyazılarında da en yüksek hata puanına sebep olan tür olmaya devam etmiştir; bu durum, incelikli anlam aktarımında kalıcı sorunların varlığını göstermektedir. Elde edilen bulgular, otomatik altyazılar, çevirmen eğitimi ve sürdürülebilirlik odaklı medya stratejileri açısından önemli çıkarımlar sunmaktadır. Çalışma ayrıca, iklim konusunda içerik üreten kurumlar için yanlış bilgilendirme riskini azaltacak ve iletişimin etkinliğini artıracak “insan merkezli iyileştirilmiş çeviri” modellerine (O’Brien, 2023) yönelmenin gerekliliğini savunmaktadır. Öte yandan, kullanıcılar açısından çeviri okuryazarlığı, özellikle de MÇ okuryazarlığı, yalnızca teknik bir beceriye değil, erişilebilir ve güvenilir iletişimi sağlamak için gerekli olan temel bir toplumsal edince karşılık gelmektedir. Bu edincin, çeviri programlarının ötesine geçen eğitim uygulamaları aracılığıyla (örn. hedefe yönelik atölyeler, ilgili derslere entegre edilen kısa modüller ve bölümler arası sertifika seçenekleri) teşvik edilmesi önerilmektedir. Böylelikle geniş kitlelerin, genel olarak MÇ aracılığıyla sunulan dijital içerikleri özel olarak da iklim konusundaki bu yolla aktarılan bilgileri eleştirel biçimde değerlendirip anlamalarına katkıda bulunabilecektir. Çalışma, MÇ değerlendirmesini çevre iletişimiyle ilişkilendirerek yüksek çeviri kalitesinin küresel iklim okuryazarlığını güçlendirmedeki ve bilinçli politika katılımını desteklemedeki rolünü vurgulamaktadır.

Anahtar Kelimeler

• makine çevirisi
• görsel-işitsel çeviri
• kalite değerlendirmesi
• makine çevirisi okuryazarlığı
• sürdürülebilirlik

Kaynakça

1. Abdelaal, N. M. (2019). Subtitling of culture-bound terms: Strategies and quality assessment. Heliyon, 5(4). https://doi.org/10.1016/j.heliyon.2019.e01532
2. Abdelaal, N. M., & Al Sarhani, A. (2021). Subtitling strategies of swear words and taboo expressions in the movie Training Day. Heliyon, 7(7). https://doi.org/10.1016/j.heliyon.2021.e07583
3. Alaa, A. M., & Al Sawi, I. (2023). The analysis and quality assessment of translation strategies in subtitling culturally specific references: Feathers. Arab World English Journal, 14(2), 155–173. https://doi.org/10.24093/awej/vol14no2.10
4. Alimen, N., Öner Bulut, S., & Karadağ, A. B. (2023). Yapay zekâ, dil ve çeviri. [AI, Language, and Translation.] In B. Küçükcan & B. F. Yıldırım (Eds.), Yapay zekâ: Disiplinlerarası yaklaşımlar [AI: Interdisciplinary Approaches] (pp. 81–103). Vakıfbank Kültür.
5. Armstrong, S., Caffrey, C., & Flanagan, M. (2006). Translating DVD subtitles from English–German and English–Japanese using example- based machine translation. In MuTra 2006 – Audiovisual Translation Scenarios: Conference Proceedings (pp. 1–10). http://www.euroconferences.info/proceedings/2006_Proceedings/200 6_Armstrong_Caffrey_Flanagan.pdf
6. Ashraf, M. (2024). Subtitling climate change: A multimodal discourse analysis study of key structural differences in the subtitles of two climate documentaries on Netflix. The International Journal of Communication and Linguistic Studies, 23(1), 163–179. https://doi.org/10.18848/2327-7882/CGP/v23i01/163-179
7. Ballantyne, A. G. (2016). Climate change communication: What can we learn from communication theory? Wiley Interdisciplinary Reviews: Climate Change, 7(3), 329–344. https://doi.org/10.1002/wcc.392
8. Bowker, L. (2020). Machine translation literacy instruction for international business students and business English instructors. Journal of Business & Finance Librarianship, 25(1–2), 25–43. https://doi.org/10.1080/08963568.2020.1794739

9. Bowker, L., & Buitrago Ciro, J. (2019). Machine translation and global research: Towards improved machine translation literacy in the scholarly community. Emerald Publishing.
10. Chadwick, A. E. (2017). Climate change communication. In D. L. Moyer (Ed.), Oxford Research Encyclopedia of Communication. Oxford University Press. https://doi.org/10.1093/acrefore/9780190228613.013.22
11. Dalmış, E., & Yıldız, M. (2025). The limits of machine translation: A quality analysis of machine translation in literary text. Uluslararası Dil, Edebiyat ve Kültür Araştırmaları Dergisi, 8(1), 314–326. https://doi.org/10.37999/udekad.1627217
12. DataReportal. (2024). Digital 2024: Global overview report. https://datareportal.com/reports/digital-2024-global-overview
13. Díaz Cintas, J. (2001). Striving for quality in subtitling: The role of a good dialogue list. Multi(media) Translation Publishing House.
14. Díaz Cintas, J. (2013). Subtitling: Theory, practice and research. In C. Millán & F. Bartrina (Eds.), The Routledge handbook of translation studies (pp. 273–287). Routledge.
15. Díaz Cintas, J. (2019). The technological turn in audiovisual translation. International Journal of Translation and Interpreting Research, 11(1), 20– 37.
16. Du, J., & Lu, J. (2024). A comparative study on the translation quality between human and machine-generated subtitles. Translation & Interpreting Studies. Advance online publication. https://doi.org/10.1075/tis.22008.du
17. Gottlieb, H. (2001). Texts, translation and subtitling—in theory, and in Denmark. In G. Anderman & M. Rogers (Eds.), Translators and translations (pp. 149–192). Multilingual Matters.
18. Hagström, H., & Pedersen, J. (2022). Subtitles in the 2020s: The influence of machine translation. Journal of Audiovisual Translation, 5(1), 110–132.
19. Haunschild, R., Bornmann, L., & Marx, W. (2016). Climate change research in view of bibliometrics. PLOS ONE, 11(7). https://doi.org/10.1371/journal.pone.0160393
20. Hulme, M. (2009). Why we disagree about climate change: Understanding controversy, inaction and opportunity. Cambridge University Press.
21. Intergovernmental Panel on Climate Change. (2013, September 27). Climate change 2013: The physical science basis [Video]. YouTube. https://www.youtube.com/watch?v=6yiTZm0y1YA (Accessed: 12 January 2025)
22. Intergovernmental Panel on Climate Change. (2022, February 28). Climate change 2022: Impacts, adaptation & vulnerability [Video]. YouTube. https://www.youtube.com/watch?v=SDRxfuEvqGg&t=160s (Accessed: 12 January 2025)
23. Intergovernmental Panel on Climate Change. (2023). AR6 synthesis report: Climate change 2023. https://www.ipcc.ch/report/ar6/syr/
24. IPCC — Intergovernmental Panel on Climate Change. (n.d.). IPCC. https://www.ipcc.ch/
25. Jin, H., & Yuan, Z. (2023). The application of machine translation in automatic dubbing in China: A case study of the feature film Mulan. Translation Horizons, 16(1), 45–59.
26. Karamitroglou, F. (1998). A proposed set of subtitling standards in Europe. Translation Journal, 2(2), 1–15.
27. Kemp, S. (2025, March 23). Digital 2024: Global Overview Report — DataReportal – Global Digital Insights. DataReportal – Global Digital Insights. https://datareportal.com/reports/digital-2024-global- overview-report
28. Koglin, A., da Silveira, J. G. P., de Matos, M. A., da Costa Silva, V. T., & Candido Moura, W. H. (2022). Quality of post-edited interlingual subtitling: FAR model, translator’s assessment and audience reception. Translation Spaces, 11(2), 275–299.
29. Lassila, H. (2019). “Dinna Fash, Sassenach”: Subtitle quality and Finnish translations of cultural references in streaming services. [Unpublished Master’s thesis], University of Turku. https://www.utupub.fi/handle/10024/147456
30. Lee, H. P. H., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025). The impact of generative AI on critical thinking: Self- reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. CHI. https://doi.org/10.1145/3706598.3713778
31. Li, M. (2024). A report on post-editing of machine translation of Climate Change 2023. Region – Educational Research and Reviews, 6(3). https://doi.org/10.32629/rerr.v6i3.1862
32. Li, S. (2022). Quality assessment of machine-translated subtitles based on the FAR model: Exemplified by The Good Wife. Journal of Audiovisual Translation Studies, 4(1), 92–110.
33. Lu, T. (2023). Applying machine translation to Chinese–English subtitling: Constraints and challenges. Linguistica Antverpiensia: New Series – Themes in Translation Studies, 22, 95-113. https://doi.org/10.52034/lans-tts.v22i.752
34. Ludera, E., Szarkowska, A., & Orrego-Carmona, D. (2024). Expertise in interlingual subtitling: Applying the FAR model to study the quality of subtitles created by professional and trainee subtitlers. Translation, Cognition & Behavior, 7(1), 86–112.
35. Metin Tekin, B. (2019). An assessment of audio-visual translation strategies in dubbing and subtitling of Frozen into Turkish. RumeliDE Dil ve Edebiyat Araştırmaları Dergisi, (15), 342–352. https://doi.org/10.29000/rumelide.580635
36. Ningtias, N. A. (2020). Subtitling strategies and qualities of flirting styles in Shakespeare’s in Love movie. [Unpublished undergraduate thesis], State Islamic Institute of Surakarta].
37. O’Brien, S. (2023). Human-centered augmented translation: against antagonistic dualisms. Perspectives, 32–32(3), 391–406. https://doi.org/10.1080/0907676X.2023.2247423
38. Okyayuz, A. Ş., & Kaya, M. (2016). A model for subtitling training. International Journal of Languages’ Education and Teaching, 4(2), 257– 275. https://doi.org/10.18298/ijlet.601
39. Öner, I., & Bengi, Z. B. (2025). Prompt, evaluate, revise: AI-assisted user manual creation in academic practice. transLogos Translation Studies Journal, 8(1), 103–347. https://doi.org/10.29228/transLogos.78
40. Öztürk Baydere, H. (2023). Rethinking translator competence in the machine translation era. transLogos Translation Studies Journal, 6(1), 45–74. https://doi.org/10.29228/transLogos.54
41. Pedersen, J. (2017). The FAR model: Assessing quality in interlingual subtitling. The Journal of Specialised Translation, 28, 210–229.
42. Popowich, F., McFetridge, P., Turcato, D., & Toole, J. (2000). Machine translation of closed captions. Machine Translation, 15(4), 311–341. https://doi.org/10.1023/A:1026724505180
43. Qiu, J., & Pym, A. (2024). Fatal flaws? Investigating the effects of machine translation errors on audience reception in the audiovisual context. Translation and Interpreting Studies. Advance online publication. https://doi.org/10.1075/tis.21012.qiu
44. Rivera-Trigueros, I. (2021). Machine translation systems and quality assessment: A systematic review. Language Resources and Evaluation, 56(2), 593–619. https://doi.org/10.1007/s10579-021-09537-5
45. Romero-Fresco, P. (2011). Subtitling through speech recognition: Respeaking. St Jerome.
46. Sixth assessment report — IPCC. (n.d.). IPCC. https://www.ipcc.ch/assessment-report/ar6/
47. Sousa, V. D. (2011). Post-editing quality in SMT subtitle translations: An analysis using BLEU. In Proceedings of the 7th Workshop on Statistical Machine Translation (pp. 32–39). Association for Computational Linguistics.
48. State of the global climate 2024. (2025, May 9). World Meteorological Organization. https://wmo.int/publication-series/state-of-global- climate-2024
49. Uslu Korkmaz, A., & Okyayuz, A. Ş. (2025). Audiovisual translation training in Turkey from the perspective of interlingual subtitling quality: Findings and suggestions. Çeviribilim ve Uygulamaları Dergisi, (TÜÇEVAS Special Issue), 109–132. https://doi.org/10.37599/ceviri.1614516
50. World Meteorological Organization. (2025). State of the global climate 2025. WMO. https://public.wmo.int/en/resources/library/state-of- global-climate-2025
51. Yan, L., Wu, Y., & Zhu, G. (2024). Automatic rating method based on deep transfer learning for machine translation considering contextual semantic awareness. Artificial Intelligence Review. Advance online publication. https://doi.org/10.1007/s10462-024-10633-x
52. Yao, G. (2022). Evaluation of machine translation in English–Chinese automatic subtitling of TED Talks. Modern Languages, Literatures, and Linguistics, 1(1), 1–17.
53. Yılmaz, M. D. (2017). Altyazı çevirilerinde kültürel öğelerin aktarımı ve çeviri sorunları [Unpublished MA thesis], İstanbul University, Institute of Social Sciences.
54. Zhou, X., & Zhou, J. (2024). A study on the quality assessment of the Chinese-English subtitle translation among four machine translation tools: A case study of Wreaths at the Foot of the Mountain. transLogos Translation Studies Journal, 7(2), 29–56. https://doi.org/10.29228/transLogos.68