Readability Transfer Capabilities of Neural Machine Translation Services

Öz

Neural Machine Translation (NMT) services demonstrate high semantic accuracy, but their ability to convey the readability of the source text is understudied. This study, therefore, provides a comprehensive evaluation of readability transfer in four leading NMT services: Amazon Translate, Azure Translator, DeepL, and Google Cloud Translation, across the English-German, English-Turkish, and German-Turkish language pairs. For this analysis, translations from various genres were assessed using a combination of language-specific readability formulas and textual metrics. Results revealed a significant directional asymmetry: readability decreased when translating from English to German or Turkish, but increased from German or Turkish to English. Statistically insignificant but consistent differences were found among the four NMT services in readability scores, with target language and source text properties having a greater influence. The findings reveal that readability is not inherently preserved in NMT and is significantly influenced by the characteristics of the target language and the nature of the source text. This highlights the critical importance of considering readability metrics alongside semantic accuracy when evaluating machine translation, especially for applications that require high accessibility or target a specific level of accessibility, suggesting potential requirements for readability-focused post-editing.

Anahtar Kelimeler

• Cross-lingual Readability
• Language Typology
• Neural Machine Translation (NMT)
• Readability
• Readability Transfer

Kaynakça

1. Amstad, T. (1978). Wie verständlich sind unsere Zeitungen? [How understandable are our newspapers?]. Studenten-Schreib-Service. google scholar
2. Ateşman, E. (1997). Türkçede okunabilirliğin ölçülmesi [Measuring readability in Turkish]. Dil Dergisi, 58, 71–74. google scholar
3. Ay, İ. E., & Duranoğlu, Y. (2022). Göz damlası prospektüslerinin okunabilirlik düzeyinin değerlendirilmesi [An evaluation of the readability of package inserts of eye drops]. Anadolu Kliniği Tıp Bilimleri Dergisi, 27(1), 55–59. https://doi.org/10.21673/ANADOLUKLIN.993863 google scholar
4. Bamberger, R., & Vanecek, E. (1984). Lesen-Verstehen-Lernen-Schreiben: Die Schwierigkeitsstufen von Texten in deutscher Sprache [Reading–understanding–learning–spelling: The degrees of difficulty of texts in German language]. Jugend und Volk. google scholar
5. Bezirci, B., & Yılmaz, A. E. (2010). Metinlerin okunabilirliğinin ölçülmesi üzerine bir yazilim kütüphanesi ve Türkçe için yeni bir okunabilirlik ölçütü [A software library for measurement of readability of texts and a new readability metric for Turkish]. Dokuz Eylül Üniversitesi Mühendislik Fakültesi Fen ve Mühendislik Dergisi, 12(3), 49–62. google scholar
6. Çıplak, G., & Balcı, A. (2022). Türkçe ders kitaplarındaki metinlerin okunabilirlik özelliğinin incelenmesi [Examining the readability of texts in Turkish textbooks]. RumeliDE Dil ve Edebiyat Araştırmaları Dergisi, 30, 170–187. https://doi.org/10.29000/RUMELIDE.1193033 google scholar
7. Cohen, S. A., Brant, A., Fisher, A. C., Pershing, S., Do, D., & Pan, C. (2024). Dr. Google vs. Dr. ChatGPT: Exploring the use of artificial intelligence in ophthalmology by comparing the accuracy, safety, and readability of responses to frequently asked patient questions regarding cataracts and cataract surgery. Seminars in Ophthalmology, 39(6), 472–479. https://doi.org/10.1080/08820538.2024.2326058 google scholar
8. Collins-Thompson, K. (2014). Computational assessment of text readability. ITL - International Journal of Applied Linguistics, 165(2), 97– 135. https://doi.org/10.1075/itl.165.2.01col google scholar

9. Crossley, S. A., Allen, D. B., & McNamara, D. S. (2011). Text readability and intuitive simplification: A comparison of readability formulas. Reading in a Foreign Language, 23(1), 84–101. google scholar
10. Dubay, W. H. (2004). The Principles of Readability. google scholar
11. Erol, H. F. (2015). Yabancı dil olarak Türkçe ders kitaplarında okunabilirlik [Readability in Turkish as a foreign language textbooks]. Türk Dili ve Edebiyatı Dergisi, 50(50), 29–38. https://dergipark.org.tr/tr/pub/iutded/issue/17078/178709 google scholar
12. Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32(3), 221–233. https://doi.org/10.1037/H0057532 google scholar
13. François, T. (2014). An analysis of a French as a Foreign language corpus for readability assessment. Proceedings of the Third Workshop on NLP for Computer-Assisted Language Learning, 107, 13–32. google scholar
14. Forcada, M. L. (2017). Making sense of neural machine translation. Translation Spaces, 6(2), 291–309. https://doi.org/10.1075/TS.6.2.06 FOR/CITE/REFWORKS google scholar
15. Gondode, P., Duggal, S., Garg, N., Lohakare, P., Jakhar, J., Bharti, S., & Dewangan, S. (2024). Comparative analysis of accuracy, readability, sentiment, and actionability: Artificial intelligence chatbots (ChatGPT and Google Gemini) versus traditional patient information leaflets for local anesthesia in eye surgery. The British and Irish Orthoptic Journal, 20(1), 183–192. https://doi.org/10.22599/BIOJ.377 google scholar
16. Gosselin, A. M., le Maux, J., & Smaili, N. (2021). Readability of accounting disclosures: A comprehensive review and research agenda. Accounting Perspectives, 20(4), 543–581. https://doi.org/10.1111/1911-3838.12275 google scholar
17. Hancı, V., Ergün, B., Gül, Ş., Uzun, Ö., Erdemir, İ., & Hancı, F. B. (2024). Assessment of readability, reliability, and quality of ChatGPT®, BARD®, Gemini®, Copilot®, Perplexity® responses on palliative care. Medicine, 103(33), e39305. https://doi.org/10.1097/MD.0000000000039305 google scholar
18. Intento. (2021). The State of Machine Translation 2021. Intento Inc. Retrieved from https://try.inten.to/machine-translation-report-2021/ google scholar
19. Işım, Ç., & Balcıoğlu, Y. S. (2023). ChatGPT: Performance of translate. 3rd International ACHARAKA Congress on Humanities and Social Sciences Proceedings Book, 47–51. google scholar
20. Jiao, W., Wang, W., Huang, J., Wang, X., Shi, S., & Tu, Z. (2023). Is ChatGPT a good translator? Yes with GPT-4 as the engine. https://docs.google.com/document/d/1GeFh6I5OrMHMI-iMFUz4e2hXhljl3xJhncFHeoAJAkg/edit?tab=t.0&usp=embed_facebook google scholar
21. Kalyoncu, M. R., & Memiş, M. (2024). Türkçe için oluşturulmuş okunabilirlik formüllerinin karşılaştırılması ve tutarlılık sorgusu [Consistency query and comparison of readability formulas created for Turkish]. Ana Dili Eğitimi Dergisi, 12(2), 417–436. www.anadiliegitimi.com google scholar
22. Kincaid, J. P., Fishburne Jr., R. P., Rogers, R. L., & Chissom, B. S. (1975). Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for navy enlisted personnel. http://library.ucf.edu google scholar
23. Loughran, T., & Mcdonald, B. (2014). Measuring readability in financial disclosures. The Journal of Finance, 69(4), 1643–1671. https://doi.org/10.1111/JOFI.12162 google scholar
24. Meyer, M. K. R., Kandathil, C. K., Davis, S. J., Durairaj, K. K., Patel, P. N., Pepper, J.-P., Spataro, E. A., & Most, S. P. (2024). Evaluation of rhinoplasty information from ChatGPT, Gemini, and Claude for readability and accuracy. Aesthetic Plastic Surgery, 49(7), 1868– 1873. https://doi.org/10.1007/S00266-024-04343-0/TABLES/3 google scholar
25. Momenaei, B., Wakabayashi, T., Shahlaee, A., Durrani, A. F., Pandit, S. A., Wang, K., Mansour, H. A., Abishek, R. M., Xu, D., Sridhar, J., Yonekawa, Y., & Kuriyan, A. E. (2023). Appropriateness and readability of ChatGPT-4-generated responses for surgical treatment of retinal diseases. Ophthalmology Retina, 7, 862–868. https://doi.org/10.1016/j.oret.2023.05.022 google scholar
26. Nygård, A. (2024). Testing teachers’ abilities to distinguish between translations produced by students, Google Translate and ChatGPT. https://doi.org/https://hdl.handle.net/11250/3148912 google scholar
27. Oliffe, M., Thompson, E., Johnston, J., Freeman, D., Bagga, H., & Wong, P. K. K. (2019). Assessing the readability and patient comprehension of rheumatology medicine information sheets: a cross-sectional health literacy study. BMJ Open, 9, :e024582. https://doi.org/10.1136/bmjopen-2018-024582 google scholar
28. Ozduran, E., Hancı, V., Erkin, Y., Özbek, İ. C., & Abdulkerimov, V. (2025). Assessing the readability, quality and reliability of responses produced by ChatGPT, Gemini, and Perplexity regarding most frequently asked keywords about low back pain. PeerJ, 13, e18847. https://doi.org/10.7717/peerj.18847 google scholar
29. Pérez-Ortiz, J. A., Forcada, M. L., & Sánchez-Martínez, F. (2022). How neural machine translation works. In D. Kenny (Ed.), Machine translation for everyone: Empowering users in the age of artificial intelligence (pp. 141–164). Language Science Press. https://doi.org/10.5281/zenodo.6760020 google scholar
30. Qi, P., Zhang, Y., Zhang, Y., Bolton, J., & Manning, C. D. (2020). Stanza : A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. https://spacy.io/ google scholar
31. Son, J., & Kim, B. (2023). Translation Performance from the User’s Perspective of Large Language Models and Neural Machine Translation Systems. Information, 14(10), 574. https://doi.org/10.3390/INFO14100574 google scholar
32. Stasimioti, M., Sosoni, V., Kermanidis, K. L., & Mouratidis, D. (2020). Machine translation quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs. Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, 441– 450. https://aclanthology.org/2020.eamt-1.47/ google scholar
33. Strzalkowski, P., Strzalkowska, A., Chhablani, J., Pfau, K., Errera, M. H., Roth, M., Schaub, F., Bechrakis, N. E., Hoerauf, H., Reiter, C., Schuster, A. K., Geerling, G., & Guthoff, R. (2024). Evaluation of the accuracy and readability of ChatGPT-4 and Google Gemini in providing information on retinal detachment: a multicenter expert comparative study. International Journal of Retina and Vitreous, 10(1), 1–11. https://doi.org/10.1186/S40942-024-00579-9/FIGURES/2 google scholar
34. Toral, A., & Way, A. (2018). What level of quality can neural machine translation attain on literary text? 263–287. https://doi.org/10.1007/ 978-3-319-91241-7_12 google scholar
35. Wang, H., Wu, H., He, Z., Huang, L., & Church, K. W. (2022). Progress in machine translation. Engineering, 18, 143–153. https://doi.org/10.1016/J.ENG.2021.03.023 google scholar
36. Wu, Y., Schuster, M., Chen, Z., Le, Q. v, Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, Ł., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., … Dean, J. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. google scholar
37. Yirmibeşoğlu, Z., Dursun, O., Dallı, H., Şahin, M., Hodzik, E., Gürses, S., & Güngör, T. (2023). Incorporating human translator style into English-Turkish literary machine translation. google scholar
38. Yılmaz, F., & Temiz, Ç. (2014). Yabancılara Türkçe öğretiminde kullanılan ders kitaplarındaki metinlerin okunabilirlik durumları [Readability conditions of texts in textbooks used in teaching Turkish to foreigners]. International Journal of Languages’ Education and Teaching, 2(1), 81–91. https://dergipark.org.tr/tr/pub/ijlet/issue/82527/1417234 google scholar
39. Zhou, S., Jeong, H., & Green, P. A. (2017). How consistent are the best-known readability equations in estimating the readability of design standards? IEEE Transactions on Professional Communication, 60(1), 97–111. https://doi.org/10.1109/TPC.2016.2635720 google scholar