Human Translator vs Machine Translation. A Case Study in Turkish-Italian Lexicography

Öz

This paper presents a comparative analysis between human translation and automated outputs within the field of Turkish-Italian lexicography. Using a corpus of example sentences extracted from Turklex—a developing online multilingual dictionary—translations were carried out manually by a professional translator and subsequently generated by four different machine translation tools: Google Translate, DeepL, Gemini, and ChatGPT-4. Particular attention is paid to the rendering of headwords, a critical element in lexicographic accuracy. By examining the divergences and overlaps across these translations, the study highlights the strengths and limitations of current NMTs and AI-driven chatbots when applied to a less commonly examined language pair. To the best of our knowledge, this is the first focused investigation into Turkish-Italian translation within a lexicographical framework. The findings aim to inform both the evaluation of machine translation tools and the broader discussion on their integration into language resource development. The research found that while ChatGPT slightly outperformed the other tools, AI chatbots and NMTs still fall short of human translators in delivering translations that are both contextually nuanced and semantically accurate.

Anahtar Kelimeler

• Machine translation
• Turkish-Italian
• Lexicography
• Online TR-IT dictionary
• NMT
• AI-driven Chatbot

Proje Numarası

--

İnsan Çevirmen ve Makine Çevirisi. Türkçe-İtalyanca Sözlükbiliminde Bir Vaka Çalışması

Öz

Bu makale, Türkçe-İtalyanca sözlükçülük alanında insan çevirisi ile otomatik çeviri çıktıları arasında karşılaştırmalı bir analiz sunmaktadır. Gelişmekte olan çok dilli çevrimiçi sözlük Turklex’ten alınan örnek cümlelerden oluşturulan bir derlem kullanılarak, çeviriler profesyonel bir çevirmen tarafından manuel olarak yapılmış, ardından dört farklı makine çeviri aracı – Google Translate, DeepL, Gemini ve ChatGPT-4 – aracılığıyla otomatik olarak üretilmiştir. Sözlükçülükte doğruluğun kritik bir unsuru olan madde başlıklarının aktarımına özel önem verilmiştir. Çeviriler arasındaki farklılıklar ve örtüşmeler incelenerek, çalışma güncel NMT (Sinirsel Makine Çevirisi) sistemleri ile yapay zeka destekli sohbet robotlarının, daha az incelenmiş dil çiftlerine uygulandığında gösterdiği güçlü ve zayıf yönleri değerlendirmektedir. Bildiğimiz kadarıyla, bu çalışma, sözlükçülük bağlamında Türkçe-İtalyanca çeviri üzerine yapılmış ilk odaklı araştırmadır. Elde edilen bulgular, hem makine çeviri araçlarının değerlendirilmesine ışık tutmayı hem de bu araçların dil kaynakları geliştirmeye entegrasyonu üzerine geniş çaplı tartışmalara katkı sağlamayı amaçlamaktadır. Araştırma, ChatGPT’nin diğer araçlardan kıyasla bir miktar daha iyi performans sergilemesine rağmen, yapay zeka destekli sohbet robotları ile NMT sistemlerinin bağlamsal olarak incelikli ve anlamsal açıdan doğru çeviriler üretme konusunda hâlen insan çevirmenlerin gerisinde kaldığını ortaya koymuştur.

Anahtar Kelimeler

• Makine Çevirisi
• Türkçe-İtalyanca
• Çevrimiçi Türkçe-İtalyanca Sözlük
• Sinirsel Makine Çevirisi (NMT)
• Yapay Zeka Destekli chatbot
• Sözlükbilim

Proje Numarası

--

Kaynakça

1. Agung, I G.A. M., Putu G. B., & Nil Wayan, S. (2024). Translation performance of Google Translate and DeepL in translating Indonesian short stories into English. In Proceedings: Linguistics, Literature, Culture and Arts International Seminar (LITERATES), 178-185.
2. Al R, Rafat, R. J., & Malkawi, M. (2025). ChatGPT translation vs. human translation: an examination of a literary text. Cogent Social Sciences, 11 (1), 1-21. DOI: 10.1080/23311886.2025.2472916.
3. Banerjee, S., & Lavie, A. (2025). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 65-72.
4. Bergenholtz, H., & Gouws, R. H. (2012). What is Lexicography?, Lexikos, 22, 31-42.
5. Bernardini, S., & Zanettin, F. (2020). When corpora meet translation: A resource for teaching, research and practice. In C. O’Brien & M. Winters (Eds.), The Routledge Handbook of Translation and Technology (pp. 383–398). London: Routledge.
6. Callison-Burch, C., Osborne, M., & Koehn, P. (2006). Re-evaluating the role of BLEU in machine translation research. In D. McCarthy, S. Wintener (Eds.), 11th Conference of the European Chapter of the Association for Computational Linguistics, Association for Cimputational Linguistics, 249-256. ChatGPT. Available from: https://chatgpt.com (February 2025).
7. Çetin, Ö., & Duran, A. (2024). A Comparative Analysis Of The Performances Of Chatgpt, Deepl, Google Translate And A Human Translator In Community Based Settings, Amasya Üniversitesi Sosyal Bilimler Dergisi (ASOBİD), 9 (15), 120-173.
8. de Schryver, G.M. (2023). Generative AI and Lexicography: The Current State of the Art Using ChatGPT, International Journal of Lexicography, XX, 1-33, https://doi.org/10.1093/ijl/ecad02. DeepL Translator. Available from: https://www.deepl.com/it/translator (February 2025).

9. D’Achille, P. (2010). L’italiano contemporaneo. Bologna: Il Mulino.
10. Dardano, M., & Trifone, P. (2003). Grammatica italiana: Con nozioni di linguistica. Bologna: Zanichelli.
11. De Mauro, T. (1999). Grande dizionario italiano dell’uso (GRADIT). Torino: UTET.
12. Demiryan, R. (2011). Dizionario. Italiano-Turco, İnkılap Kitabevi.
13. Deng, J., & Lin, Y. (2022). The Benefits and Challenges of ChatGPT: An Overview, Frontiers in Computing and Intelligent Systems, 2(2), 81-83, http://dx.doi.org/10.54097/fcis.v2i2.4465.
14. Deng, Y. (2024). ChatGPT-based Research on the Influence of AI on English Translation, Applied Mathematics and Nonlinear Sciences, 9(1), 1-14, DOI:10.2478/amns-2024-2044.
15. Fan, P., & Gong, X.H. (2023). The Application of ChatGPT in Translation Teaching: Changes, Challenges, and Responses. International Journal of Education and Humanities 11(2). 49-52 DOI:10.54097/ijeh.v11i2.13530.
16. Fuchs, K. (2023). Exploring the opportunities and challenges of NLP models in higher education: Is Chat GPT a blessing or a curse?, Frontiers in Education, 8. https://doi.org/10.3389/feduc.2023.1166682. Gemini AI. Available from https://gemini.google.com/app?hl=tr (February 2025).
17. Ghosh, S., & Caliskan, A. (2023). ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languages, ArXiv, arXiv:2305.10510. Google Translate. Available from https://translate.google.com.tr (February 2025).
18. Güneş, S. (2018). İtalyanca ve Türkçe Belirsiz Nicelik İfadelerinde Adların Kullanımı Üzerine Bir İnceleme, RumeliDE Journal of Language and Literature Studies, 214-231.
19. Gürlek, M. (2023). Türkçe Öğrenici Sözlüğü için Bir Maddeyi Hazırlamak, Türkiyat Mecmuasi, 33(2), 625-640, DOI: 10.26650/iuturkiyat.1306809.
20. Habeeb, L. S. (2020). Investigating the Effectiveness of Google Translate among Iraqui Students, International Journal of Innovation, Creativity and Change, 12(12), 104-105.
21. Jiao, W., Wang, H., Wenxuan, J., Wang, X., Shi, S., & Tu, Z. (2023). Is Chat GPT A Good Translator? Yes With GPT-4As the Engine, arXiv, 1-10, https://arxiv.org/pdf/2301.08745.
22. Karnal, A.R., Vanmacher Pereira, V. (2015). Reading Strategies in a L2: A Study on Machine Translation, The Reading Matrix: An International Online Journal, 15 (2), 1-11.
23. Kaya, R. (2020). İtalyanlarin Türkçe Öğretimi: Harfler Ve Ses Karşiliklari, Hacettepe Üniversitesi Yabancı Dil Olarak Türkçe Araştırmaları Dergisi, 6, 51-75.
24. Kembaren, F. R. W., Hasibuan, A. K., & Natasya, A. (2023). Technology Trends in Translation: A Comparative Analysis of Machine and Human Translation, Absorbent Mind. Journal of Psychology and Child Development, 3 (2), 168-183, 10.37680/absorbent_mind.v3i2.4486.
25. Khoong, E. C., Steinbrook, E., & Brown, C. (2019). Assessing the Use of google Translate for Spanish and Chinese Translations of Emergency Department Discharge Instructions, JAMA Intern Med., 179 (4), 580-582. doi:10.1001/jamainternmed.2018.7653.
26. Khoshafah, F. (2023). ChatGPT for Arabic-English Translation: Evaluating Accuracy, Research Square,1-18, https://doi.org/10.21203/rs.3.rs-2814154/v2.
27. Kline, R. B. (2011). Principles and practice of structural equation modeling. London: Guilford publications.
28. Kocmi, T., & Federman, C. (2023). Large Language Models Are State-of-the-Art Evaluators of Translation Quality, Proceedings of the 24th Annual Conference of the European Association for Machine Translation, European Association for Machine Translation, 193-203.
29. Lew, R. (2023). ChatGPT as a COBUILD lexicographer, Humanities and Social Communications, 10, 704. https://doi.org/10.1057/s41599-023-02119-6.
30. Lew, R. (2024). Dictionaries and lexicography in the AI era, Humanit Soc Sci Commun, 11, 426. https://doi.org/10.1057/s41599-024-02889-7.
31. Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger. J. (2024). ChatGPT in healthcare: A taxonomy and systematic review, Computer Methods and Programs in Biomedicine, 245. https://doi.org/10.1016/j.cmpb.2024.108013.
32. Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., Wu, Z., Zhao, L., Zhu, D., Li, X., Qiang, N., Shen, D., Liu, T., & Ge, B. (2023). Summary of ChatGPT/GPT-4 research and perspective towards the future of large language models. arXiv:2304.01852. https://doi.org/10.1016/j.metrad.2023.100017.
33. Lo Cascio, V. (2003). Grammatica dell’italiano per stranieri. Firenze: Le Monnier.
34. Naveen, P., & Trojovsky, P. (2024). Overview and Challenges of Machine Translation for Contextually Appropriate Translations, iScience, 27 (10), 1-25, DOI:10.1016/j.isci.2024.110878.
35. Manzelli, G. (2020). Il contributo dell’italiano al lessico ludico del turco di Turchia. Italiano LinguaDue, 2, 82-97.
36. Marello, C. (1989). Dizionari bilingui: Studi di metalessicografia. Alessandria: Edizioni dell’Orso.
37. Marello, C. (2015). La lessicografia bilingue in Italia oggi. In R. Simone (Ed.), Enciclopedia dell’Italiano (pp. 1–6). Roma: Istituto dell’Enciclopedia Italiana Treccani.
38. Özkan, N. (2020). L’italiano nel turco, Italiano LinguaDue, 2, 98-106.
39. Rocchi, L. (2020). Gli elementi turco-ottomani nel Vocabolario italiano, e kurdo di Maurizio Garzoni (1887) e la loro importanza storico-documentaria, Rivista internazionale di Tecnica della Traduzione / International Journal of Translation, 22, 231-249.
40. Rossi, F. (2005). Lessicografia e traduzione: Problemi di equivalenza tra italiano e lingue europee. Torino: UTET.
41. Rundell, M. (2024). Automating the Creation of Dictionaries: Are We Nearly There?, Proceedings of the 16th International Conference of the Asian Association for Lexicography:“Lexicography, Artificial Intelligence, and Dictionary Users”. Seoul: Yonsei University, 1–9.
42. Sahari, Y., Al-Kadi Talib, A.M., & Ali, J. K. M. (2023). A Cross Sectional Study of ChatGPT in Translation: Magnitude of Use, Attitudes, and Uncertainties, J Psycholinguist Res, 52, 2937–2954, https://doi.org/10.1007/s10936-023-10031-y.
43. Siu, S. C. (2023). ChatGPT and GPT-4 for Professional Translators: Exploring the Potential of Large Language Models in Translation, SSRN, 1-36, https://dx.doi.org/10.2139/ssrn.4448091.
44. Serianni, L. (2005). Italiano: Grammatica, sintassi, dubbi. Milano: Garzanti.
45. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A study of translation edit rate with targeted human annotation, In Proceedings of Association for Machine Translation in the Americas (AMTA), 223-231.
46. Son, J., & Kim, B., (2023). Translation Performance from the User’s Perspective of Large Language Models and Neural Machine Translation Systems, Information, 14 (10), 574, https://doi.org/10.3390/info14100574.
47. Sungran, K. (2023). An Analysis of ChatGPT’s Language Translation Based on the Korean Film Minari, Journal of English Teaching through Movies and Media, 24 (4), 1-14 doi: https://doi.org/10.16875/stem.2023.24.4.1.
48. Tan, R., Long, X., & Bamigbade, O. O. (2023). Comparative Research on Machine Translation and Human Translation of Examples in Dictionary from the Perspective of Skops Theory, IEIT 2023, AHSSEH 10, 342–353, https://doi.org/10.2991/978-94-6463-230-9_41.
49. Yilmaz, E. D., Naumovska, I., & Aggarwal, A. V. (2023). AI-Driven Labor Substitution: Evidence from Google Translate and ChatGPT. INSEAD. http://ssrn.com/abstract=4400516.
50. Xiu, P., & Xeauyin, L. (2018). Human translation vs machine translation: The practitioner phenomenology, Linguistics and Culture Review, 2(1), 13-23, https://doi.org/10.37028/lingcure.v2n1.8.
51. Zanettin, F., Bernardini, S., & Stewart, D. (2023). Corpora in Translator Education. London: Routledge.