RAG'de Dil Uçurumunu Köprüleme: Türkçe Erişim ve Üretim Üzerine Bir Vaka Çalışması

Yıl 2025, Cilt: 05 Sayı: 01, 38 - 49, 31.07.2025

Erdoğan Bikmaz , Mohammed Briman , Serdar Arslan

https://izlik.org/JA42PW55KK

Öz

Büyük Dil Modelleri (BDM'ler) ve BDM tabanlı RAG sistemlerinin yükselişiyle birlikte, çok dilli ortamlarda yoğun metin sistemlerini ele almak için BDM'lerin akıl yürütme yeteneklerini kullanan RAG uygulamaları geliştirmeye yönelik yüksek bir talep bulunmaktadır. Ancak, RAG bileşenleri öncelikle İngilizce dili için geliştirilmiştir, bu da özellikle Türkçe için BDM'lerin cevaplayabilmesi amacıyla kesin çok dilli bilgiyiRetrieval ve yapılandırma yeteneklerini engellemektedir. Bu çalışmada, Türkçe soru-cevapRetrieval ve üretme görevlerini ele alan kapsamlı RAG sistemleri geliştirmenin etkilerini araştırmayı amaçlıyoruz. Veri alımı veRetrieval için kullanılan gömütme modeli ve bir sorguyla alâkalılıklarına göreRetrieval edilen belgeleri sıralayan bir yeniden sıralayıcı model olmak üzere iki ana bileşeni Türkçe veriler üzerinde ince ayarlayarak deneyler yapıyoruz. Dört RAG sistemini altı değerlendirme metriği kullanarak değerlendiriyoruz. Deneysel sonuçlar,Retrieval bileşenlerinin Türkçe veriler üzerinde ince ayarlanmasının BDM yanıtlarının doğruluğunu artırdığını ve iyileştirilmiş bağlam oluşturmaya yol açtığını göstermektedir.

Anahtar Kelimeler

bilgi erişimi destekli üretim , büyük dil modelleri , gömütme , bilgi erişimi

Bridging the Language Gap in RAG: A Case Study on Turkish Retrieval and Generation

https://izlik.org/JA42PW55KK

Öz

With the rise of Large Language Models (LLMs) and LLM-based RAG systems, there is a high demand for developing RAG applications that utilize LLM reasoning capabilities for handling intensive text systems in multilingual settings. However, RAG components are primarily developed for the English language, which hinders their ability to retrieve and construct precise multilingual information for LLMs to answer, especially for the Turkish language. In this work, we aim to explore the effects of developing comprehensive RAG systems that handle Turkish question-answer retrieval and generation tasks. We experiment with fine-tuning two major components on Turkish data: the embedding model used for data ingestion and retrieval, and a reranker model that ranks the retrieved documents based on their relevance to a query. We evaluate four RAG systems using six evaluation metrics. Experimental results show that fine-tuning retrieval components on Turkish data improves the accuracy of LLM responses and leads to improved context construction.

Anahtar Kelimeler

retrieval augmented generation , large language models , embedding , information retrieval

Kaynakça

• [1] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L ., Polosukhin, I.: Attention is All you Need. In: Conference on Neural Information Processing Systems (NeurIPS) (2017). https://papers.nips.cc/paperfiles/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa− Abstract.html
• [2] OpenAI: GPT-4 Technical Report. arXiv preprint (2023). https://arxiv.org/abs/ 2303.08774
• [3] Unknown: Introducing Claude 3.5 Sonnet. https://www.anthropic.com/news/ claude-3-5-sonnet. No date provided in citation (n.d.)
• [4] Grattafiori: The Llama 3 herd of models. arXiv preprint (2024). https://arxiv. org/abs/2407.21783
• [5] Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., De Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L.R., Lachaux, M.A., Stock, P., Scao, T.L., Lavril, T., Wang, T., Lacroix, T., Sayed, W.E.: Mistral 7B. arXiv preprint (2023). https://arxiv.org/abs/2310. 06825
• [6] Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R.J., Javaheripi, M., Kauffmann, P., Lee, J.R., Lee, Y.T., Li, Y., Liu, W., Mendes, C.C.T., Nguyen, A., Price, E., Gustavo, D.R., Saarikivi, O., Salim, A., Shah, S., Wang, X., Ward, R., Wu, Y., Yu, D., Zhang, C., Zhang, Y.: PHI-4 Technical Report. arXiv preprint (2024). https://arxiv.org/abs/2412.08905
• [7] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D.: Chain-of-Thought prompting elicits reasoning in large language models. arXiv preprint (2022). https://arxiv.org/abs/2201.11903
• [8] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T.L., Cao, Y., Narasimhan, K.: Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv preprint (2023). https://arxiv.org/abs/2305.10601
• [9] Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., Liu, T.: A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Office Information Systems (2024) arXiv:2311.05232. Preprint available at https://arxiv.org/abs/2311.05232
• [10] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Ku¨ttler, H., Lewis, M., Yih, W.T., Rockt¨aschel, T., Riedel, S., Kiela, D.: Retrieval-Augmented Generation for Knowledge-Intensive NLP tasks. arXiv preprint (2020). https:// arxiv.org/abs/2005.11401
• [11] Reimers, N., Gurevych, I.: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv preprint (2019). https://arxiv.org/abs/1908.10084
• [12] Jegou, H., Douze, M., Johnson, J.: Faiss: A library for efficient similarity search. Facebook Engineering blog. Published March 29, 2017, but cited as 2018 (2018). https://engineering.fb.com/2017/03/29/data-infrastructure/ faiss-a-library-for-efficient-similarity-search/
• [13] Unknown: What is Semantic Search? https://cohere.com/llmu/ what-is-semantic-search. No date provided in citation (n.d.)
• [14] Ahmad, S.R.: Enhancing multilingual information retrieval in mixed human resources environments: a RAG model implementation for multicultural enterprise. arXiv preprint (2024). https://arxiv.org/abs/2401.01511
• [15] Ustu¨n, A., Aryabumi, V., Yong, Z.X., Ko, W.Y., D’souza, D., Onilude, G., Bhan-¨ dari, N., Singh, S., Ooi, H.L., Kayid, A., Vargus, F., Blunsom, P., Longpre, S., Muennighoff, N., Fadaee, M., Kreutzer, J., Hooker, S.: Aya model: An Instruction finetuned Open-Access Multilingual language model. arXiv preprint (2024). https://arxiv.org/abs/2402.07827
• [16] Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., al.: QWEN2 Technical Report. arXiv preprint (2024). https://arxiv. org/abs/2407.10671
• [17] Chirkova, N., Rau, D., D´ejean, H., Formal, T., Clinchant, S., Nikoulina, V.: Retrieval-augmented generation in multilingual settings. arXiv preprint (2024). https://arxiv.org/abs/2407.01463
• [18] Chen, W.H.C.S.Y.W.M.K.Z.X.Z.H..Y.D. T.: Dense X retrieval: What retrieval granularity should we use? arXiv preprint (2023). https://arxiv.org/abs/2312. 06648
• [19] Gumma, V., Raghunath, A., Jain, M., Sitaram, S.: HEALTH-PARIKSHA: Assessing RAG models for health chatbots in Real-World multilingual settings. arXiv preprint (2024). https://arxiv.org/abs/2410.13671
• [20] Xu, P., Wu, H., Wang, J., Lin, R., Tan, L.: Traditional Chinese Medicine Case Analysis System for High-Level Semantic Abstraction: Optimized with Prompt and RAG. arXiv preprint (2024). https://arxiv.org/abs/2411.15491
• [21] Zhang, T., Patil, S.G., Jain, N., Shen, S., Zaharia, M., Stoica, I., Gonzalez, J.E.: RAFT: Adapting Language Model to Domain Specific RAG. arXiv preprint (2024). https://arxiv.org/abs/2403.10131
• [22] Zhang, T., Patil, S.G., Jain, N., Shen, S., Zaharia, M., Stoica, I., Gonzalez, J.E.: RAFT: Adapting Language Model to Domain Specific RAG. arXiv preprint. This uses the same source as the ’raft’ entry, labelled ’chunking’ in the prompt list. (2024). https://arxiv.org/abs/2403.10131
• [23] Chen, X., Cardie, C.: Unsupervised multilingual word embeddings. arXiv preprint (2018). https://arxiv.org/abs/1808.08933
• [24] Yang, Y., Cer, D., Ahmad, A., Guo, M., Law, J., Constant, N., Abrego, G.H., Yuan, S., Tar, C., Sung, Y.H., Strope, B., Kurzweil, R.: Multilingual universal sentence encoder for semantic retrieval. arXiv preprint (2019). https://arxiv.org/ abs/1907.04307
• [25] Muennighoff, N., Tazi, N., Magne, L., Reimers, N.: MTEB: Massive Text Embedding Benchmark. arXiv preprint (2022). https://arxiv.org/abs/2210.07316
• [26] Unknown: MTEB: Massive Text Embedding Benchmark. https://huggingface.co/ blog/mteb. No date provided in citation (n.d.)
• [27] Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: MINILM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. arXiv preprint (2020). https://arxiv.org/abs/2002.10957
• [28] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint (2015). https://arxiv.org/abs/1503.02531
• [29] Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., Zhang, M.: Towards General Text Embeddings with Multi-stage Contrastive Learning. arXiv preprint (2023). https://arxiv.org/abs/2308.03281
• [30] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint (2018). https://arxiv.org/abs/1810.04805
• [31] Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., Wei, F.: Text embeddings by Weakly-Supervised contrastive pre-training. arXiv preprint (2022). https://arxiv.org/abs/2212.03533
• [32] Hamed, Z., Mostafa, D., Bruce, C.W., Erik, L.-M., Jaap, K.: From Neural ReRanking to Neural Ranking learning a sparse representation for inverted indexing. ACM Proceedings, 497–506 (2018)
• [33] Khattab, O., Zaharia, M.: ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. arXiv preprint (2020). https://arxiv. org/abs/2004.12832
• [34] De Souza P Moreira, G., Ak, R., Schifferer, B., Xu, M., Osmulski, R., Oldridge, E.: Enhancing QA Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAG (2024). https://arxiv.org/abs/2409.07691
• [35] Kesgin, H.T., Yuce, M.K., Amasyali, M.F.: Developing and evaluating tiny to medium-sized turkish bert models. arXiv preprint arXiv:2307.14134 (2023)
• [36] ytu-ce-cosmos/turkish-colbert · Hugging Face. https://huggingface.co/ ytu-ce-cosmos/turkish-colbert
• [37] Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Tiwary, S., Wang, T.: MS MARCO: A Human Generated MAchine Reading COmprehension Dataset (2016). https://arxiv.org/abs/1611.09268
• [38] MeTin/WIKIRAG-TR · Datasets at Hugging Face. https://huggingface.co/ datasets/Metin/WikiRAG-TR
• [39] Gu¨nther, M., Ong, J., Mohr, I., Abdessalem, A., Abel, T., Akram, M.K., Guzman, S., Mastrapas, G., Sturua, S., Wang, B., Werk, M., Wang, N., Xiao, H.: JINA Embeddings 2: 8192-Token General-Purpose text embeddings for long documents (2023). https://arxiv.org/abs/2310.19923
• [40] Robinson, J., Chuang, C.-Y., Sra, S., Jegelka, S.: Contrastive Learning with Hard Negative Samples (2020). https://arxiv.org/abs/2010.04592
• [41] Thakur, N., Reimers, N., Ru¨ckl´e, A., Srivastava, A., Gurevych, I.: BEIR: a heterogenous benchmark for zero-shot evaluation of information retrieval models (2021). https://arxiv.org/abs/2104.08663
• [42] Es, S., James, J., Espinosa-Anke, L., Schockaert, S.: RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv preprint (2023). https://arxiv. org/abs/2309.15217
• [43] Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Proceedings of the Meeting of the Association for Computational Linguistics, Barcelona, Spain, pp. 74–81 (2004)
• [44] Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: Evaluating Text Generation with BERT. arXiv preprint (2019). https://arxiv.org/abs/ 1904.09675
• [45] LangChain: Recursively split by character. https://python.langchain.com/v0.1/ docs/modules/data connection/document transformers/recursive text splitter/. No date provided in citation (n.d.)
• [46] Gemini Team: Gemini: a family of highly capable multimodal models. arXiv preprint (2023). https://arxiv.org/abs/2312.11805
• [47] Team, G., et. al., Gemma 3 Technical Report (2025). https://arxiv.org/abs/2503.19786
• [48] Amati, G.: BM25, pp. 257–260 (2009). https://doi.org/10.1007/ 978-0-387-39940-9\{ 9 . https://doi.org/10.1007/978-0-387-39940-9921
• [49] Chroma-Core: GitHub - chroma-core/chroma: the AI-native open-source embedding database. https://github.com/chroma-core/chroma
• [50] sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 · Hugging Face. https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2