RAG'de Dil Uçurumunu Köprüleme: Türkçe Erişim ve Üretim Üzerine Bir Vaka Çalışması

Öz

Büyük Dil Modelleri (BDM'ler) ve BDM tabanlı RAG sistemlerinin yükselişiyle birlikte, çok dilli ortamlarda yoğun metin sistemlerini ele almak için BDM'lerin akıl yürütme yeteneklerini kullanan RAG uygulamaları geliştirmeye yönelik yüksek bir talep bulunmaktadır. Ancak, RAG bileşenleri öncelikle İngilizce dili için geliştirilmiştir, bu da özellikle Türkçe için BDM'lerin cevaplayabilmesi amacıyla kesin çok dilli bilgiyiRetrieval ve yapılandırma yeteneklerini engellemektedir. Bu çalışmada, Türkçe soru-cevapRetrieval ve üretme görevlerini ele alan kapsamlı RAG sistemleri geliştirmenin etkilerini araştırmayı amaçlıyoruz. Veri alımı veRetrieval için kullanılan gömütme modeli ve bir sorguyla alâkalılıklarına göreRetrieval edilen belgeleri sıralayan bir yeniden sıralayıcı model olmak üzere iki ana bileşeni Türkçe veriler üzerinde ince ayarlayarak deneyler yapıyoruz. Dört RAG sistemini altı değerlendirme metriği kullanarak değerlendiriyoruz. Deneysel sonuçlar,Retrieval bileşenlerinin Türkçe veriler üzerinde ince ayarlanmasının BDM yanıtlarının doğruluğunu artırdığını ve iyileştirilmiş bağlam oluşturmaya yol açtığını göstermektedir.

Anahtar Kelimeler

• bilgi erişimi destekli üretim
• büyük dil modelleri
• gömütme
• bilgi erişimi

Bridging the Language Gap in RAG: A Case Study on Turkish Retrieval and Generation

Öz

With the rise of Large Language Models (LLMs) and LLM-based RAG systems, there is a high demand for developing RAG applications that utilize LLM reasoning capabilities for handling intensive text systems in multilingual settings. However, RAG components are primarily developed for the English language, which hinders their ability to retrieve and construct precise multilingual information for LLMs to answer, especially for the Turkish language. In this work, we aim to explore the effects of developing comprehensive RAG systems that handle Turkish question-answer retrieval and generation tasks. We experiment with fine-tuning two major components on Turkish data: the embedding model used for data ingestion and retrieval, and a reranker model that ranks the retrieved documents based on their relevance to a query. We evaluate four RAG systems using six evaluation metrics. Experimental results show that fine-tuning retrieval components on Turkish data improves the accuracy of LLM responses and leads to improved context construction.

Anahtar Kelimeler

• retrieval augmented generation
• large language models
• embedding
• information retrieval

Kaynakça

1. [1] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L ., Polosukhin, I.: Attention is All you Need. In: Conference on Neural Information Processing Systems (NeurIPS) (2017). https://papers.nips.cc/paperfiles/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa− Abstract.html
2. [2] OpenAI: GPT-4 Technical Report. arXiv preprint (2023). https://arxiv.org/abs/ 2303.08774
3. [3] Unknown: Introducing Claude 3.5 Sonnet. https://www.anthropic.com/news/ claude-3-5-sonnet. No date provided in citation (n.d.)
4. [4] Grattafiori: The Llama 3 herd of models. arXiv preprint (2024). https://arxiv. org/abs/2407.21783
5. [5] Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., De Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L.R., Lachaux, M.A., Stock, P., Scao, T.L., Lavril, T., Wang, T., Lacroix, T., Sayed, W.E.: Mistral 7B. arXiv preprint (2023). https://arxiv.org/abs/2310. 06825
6. [6] Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R.J., Javaheripi, M., Kauffmann, P., Lee, J.R., Lee, Y.T., Li, Y., Liu, W., Mendes, C.C.T., Nguyen, A., Price, E., Gustavo, D.R., Saarikivi, O., Salim, A., Shah, S., Wang, X., Ward, R., Wu, Y., Yu, D., Zhang, C., Zhang, Y.: PHI-4 Technical Report. arXiv preprint (2024). https://arxiv.org/abs/2412.08905
7. [7] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D.: Chain-of-Thought prompting elicits reasoning in large language models. arXiv preprint (2022). https://arxiv.org/abs/2201.11903
8. [8] Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T.L., Cao, Y., Narasimhan, K.: Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv preprint (2023). https://arxiv.org/abs/2305.10601

9. [9] Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., Liu, T.: A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Office Information Systems (2024) arXiv:2311.05232. Preprint available at https://arxiv.org/abs/2311.05232
10. [10] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Ku¨ttler, H., Lewis, M., Yih, W.T., Rockt¨aschel, T., Riedel, S., Kiela, D.: Retrieval-Augmented Generation for Knowledge-Intensive NLP tasks. arXiv preprint (2020). https:// arxiv.org/abs/2005.11401
11. [11] Reimers, N., Gurevych, I.: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv preprint (2019). https://arxiv.org/abs/1908.10084
12. [12] Jegou, H., Douze, M., Johnson, J.: Faiss: A library for efficient similarity search. Facebook Engineering blog. Published March 29, 2017, but cited as 2018 (2018). https://engineering.fb.com/2017/03/29/data-infrastructure/ faiss-a-library-for-efficient-similarity-search/
13. [13] Unknown: What is Semantic Search? https://cohere.com/llmu/ what-is-semantic-search. No date provided in citation (n.d.)
14. [14] Ahmad, S.R.: Enhancing multilingual information retrieval in mixed human resources environments: a RAG model implementation for multicultural enterprise. arXiv preprint (2024). https://arxiv.org/abs/2401.01511
15. [15] Ustu¨n, A., Aryabumi, V., Yong, Z.X., Ko, W.Y., D’souza, D., Onilude, G., Bhan-¨ dari, N., Singh, S., Ooi, H.L., Kayid, A., Vargus, F., Blunsom, P., Longpre, S., Muennighoff, N., Fadaee, M., Kreutzer, J., Hooker, S.: Aya model: An Instruction finetuned Open-Access Multilingual language model. arXiv preprint (2024). https://arxiv.org/abs/2402.07827
16. [16] Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., al.: QWEN2 Technical Report. arXiv preprint (2024). https://arxiv. org/abs/2407.10671
17. [17] Chirkova, N., Rau, D., D´ejean, H., Formal, T., Clinchant, S., Nikoulina, V.: Retrieval-augmented generation in multilingual settings. arXiv preprint (2024). https://arxiv.org/abs/2407.01463
18. [18] Chen, W.H.C.S.Y.W.M.K.Z.X.Z.H..Y.D. T.: Dense X retrieval: What retrieval granularity should we use? arXiv preprint (2023). https://arxiv.org/abs/2312. 06648
19. [19] Gumma, V., Raghunath, A., Jain, M., Sitaram, S.: HEALTH-PARIKSHA: Assessing RAG models for health chatbots in Real-World multilingual settings. arXiv preprint (2024). https://arxiv.org/abs/2410.13671
20. [20] Xu, P., Wu, H., Wang, J., Lin, R., Tan, L.: Traditional Chinese Medicine Case Analysis System for High-Level Semantic Abstraction: Optimized with Prompt and RAG. arXiv preprint (2024). https://arxiv.org/abs/2411.15491
21. [21] Zhang, T., Patil, S.G., Jain, N., Shen, S., Zaharia, M., Stoica, I., Gonzalez, J.E.: RAFT: Adapting Language Model to Domain Specific RAG. arXiv preprint (2024). https://arxiv.org/abs/2403.10131
22. [22] Zhang, T., Patil, S.G., Jain, N., Shen, S., Zaharia, M., Stoica, I., Gonzalez, J.E.: RAFT: Adapting Language Model to Domain Specific RAG. arXiv preprint. This uses the same source as the ’raft’ entry, labelled ’chunking’ in the prompt list. (2024). https://arxiv.org/abs/2403.10131
23. [23] Chen, X., Cardie, C.: Unsupervised multilingual word embeddings. arXiv preprint (2018). https://arxiv.org/abs/1808.08933
24. [24] Yang, Y., Cer, D., Ahmad, A., Guo, M., Law, J., Constant, N., Abrego, G.H., Yuan, S., Tar, C., Sung, Y.H., Strope, B., Kurzweil, R.: Multilingual universal sentence encoder for semantic retrieval. arXiv preprint (2019). https://arxiv.org/ abs/1907.04307
25. [25] Muennighoff, N., Tazi, N., Magne, L., Reimers, N.: MTEB: Massive Text Embedding Benchmark. arXiv preprint (2022). https://arxiv.org/abs/2210.07316
26. [26] Unknown: MTEB: Massive Text Embedding Benchmark. https://huggingface.co/ blog/mteb. No date provided in citation (n.d.)
27. [27] Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: MINILM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. arXiv preprint (2020). https://arxiv.org/abs/2002.10957
28. [28] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint (2015). https://arxiv.org/abs/1503.02531
29. [29] Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., Zhang, M.: Towards General Text Embeddings with Multi-stage Contrastive Learning. arXiv preprint (2023). https://arxiv.org/abs/2308.03281
30. [30] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint (2018). https://arxiv.org/abs/1810.04805
31. [31] Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., Wei, F.: Text embeddings by Weakly-Supervised contrastive pre-training. arXiv preprint (2022). https://arxiv.org/abs/2212.03533
32. [32] Hamed, Z., Mostafa, D., Bruce, C.W., Erik, L.-M., Jaap, K.: From Neural ReRanking to Neural Ranking learning a sparse representation for inverted indexing. ACM Proceedings, 497–506 (2018)
33. [33] Khattab, O., Zaharia, M.: ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. arXiv preprint (2020). https://arxiv. org/abs/2004.12832
34. [34] De Souza P Moreira, G., Ak, R., Schifferer, B., Xu, M., Osmulski, R., Oldridge, E.: Enhancing QA Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAG (2024). https://arxiv.org/abs/2409.07691
35. [35] Kesgin, H.T., Yuce, M.K., Amasyali, M.F.: Developing and evaluating tiny to medium-sized turkish bert models. arXiv preprint arXiv:2307.14134 (2023)
36. [36] ytu-ce-cosmos/turkish-colbert · Hugging Face. https://huggingface.co/ ytu-ce-cosmos/turkish-colbert
37. [37] Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Tiwary, S., Wang, T.: MS MARCO: A Human Generated MAchine Reading COmprehension Dataset (2016). https://arxiv.org/abs/1611.09268
38. [38] MeTin/WIKIRAG-TR · Datasets at Hugging Face. https://huggingface.co/ datasets/Metin/WikiRAG-TR
39. [39] Gu¨nther, M., Ong, J., Mohr, I., Abdessalem, A., Abel, T., Akram, M.K., Guzman, S., Mastrapas, G., Sturua, S., Wang, B., Werk, M., Wang, N., Xiao, H.: JINA Embeddings 2: 8192-Token General-Purpose text embeddings for long documents (2023). https://arxiv.org/abs/2310.19923
40. [40] Robinson, J., Chuang, C.-Y., Sra, S., Jegelka, S.: Contrastive Learning with Hard Negative Samples (2020). https://arxiv.org/abs/2010.04592
41. [41] Thakur, N., Reimers, N., Ru¨ckl´e, A., Srivastava, A., Gurevych, I.: BEIR: a heterogenous benchmark for zero-shot evaluation of information retrieval models (2021). https://arxiv.org/abs/2104.08663
42. [42] Es, S., James, J., Espinosa-Anke, L., Schockaert, S.: RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv preprint (2023). https://arxiv. org/abs/2309.15217
43. [43] Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Proceedings of the Meeting of the Association for Computational Linguistics, Barcelona, Spain, pp. 74–81 (2004)
44. [44] Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: Evaluating Text Generation with BERT. arXiv preprint (2019). https://arxiv.org/abs/ 1904.09675
45. [45] LangChain: Recursively split by character. https://python.langchain.com/v0.1/ docs/modules/data connection/document transformers/recursive text splitter/. No date provided in citation (n.d.)
46. [46] Gemini Team: Gemini: a family of highly capable multimodal models. arXiv preprint (2023). https://arxiv.org/abs/2312.11805
47. [47] Team, G., et. al., Gemma 3 Technical Report (2025). https://arxiv.org/abs/2503.19786
48. [48] Amati, G.: BM25, pp. 257–260 (2009). https://doi.org/10.1007/ 978-0-387-39940-9\{ 9 . https://doi.org/10.1007/978-0-387-39940-9921
49. [49] Chroma-Core: GitHub - chroma-core/chroma: the AI-native open-source embedding database. https://github.com/chroma-core/chroma
50. [50] sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 · Hugging Face. https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2