İlkokul 3. Sınıf Matematik Eğitiminde Halüsinasyonların Azaltılması: Yerel RAG ve Büyük Dil Modellerinin Ampirik Bir Analizi

Abstract

Bu çalışma, Türkçe ilkokul matematik eğitiminde karşılaşılan güncellik ve halüsinasyon (uydurma) sorunlarını gidermek amacıyla, tamamen yerel donanım üzerinde çalışan optimize edilmiş bir RAG mimarisini incelemektedir. Araştırma kapsamında; Llama3, Mistral, Gemma ve Phi3 dil modelleri; MiniLM ve BERTurk gömme modelleriyle birlikte uçtan uca test edilmiştir. Çalışmanın temel verisi, toplamda yaklaşık 20 saatlik içerik sunan 25 eğitim videosundan elde edilen ve iki alan uzmanı tarafından oluşturulan 1.270 satırlık bir "Soru-Cevap-Bağlam" altın standart veri setidir. Veri setinin güvenilirliği, 0.85 Cohen’s Kappa katsayısı ile doğrulanmıştır. Değerlendirme sürecinde, üretilen yanıtların ilgi düzeyi ve sadakatini ölçmek için 1–5 Likert ölçeği üzerinden yapılandırılmış bir "LLM-as-a-Judge" yaklaşımıyla Google Gemini kullanılmıştır. Deneysel bulgular, RAG mimarisinin Mistral modelindeki halüsinasyon oranını %60'tan %35'e düşürerek güvenilirliği anlamlı derecede artırdığını göstermektedir. Gemma-7B modeli; ~2.4 saniyelik yanıt süresi ve 0.28 F1 skoru ile en başarılı sistem olarak öne çıkmıştır.

Keywords

• Erişimle Desteklenmiş Metin Üretimi
• Büyük Dil Modelleri
• Doğal Dil İşleme
• Halüsinasyon
• Eğitim Teknolojileri
• Yerel Büyük Dil Modelleri

Mitigating Hallucinations in Third-Grade Primary Mathematics Education: An Empirical Analysis of Local RAG and LLMs

Abstract

This study investigates an optimized Retrieval-Augmented Generation (RAG) architecture running entirely on local hardware to address the issues of recency and hallucination encountered in Turkish primary school mathematics education. Within the scope of the research, Llama3, Mistral, Gemma, and Phi3 language models were tested end-to-end using fixed and semantic chunking strategies alongside multilingual MiniLM and Turkish-specific BERTurk embedding models. The primary dataset comprises texts extracted from 25 educational videos, totaling approximately 20 hours of content, which were curated into a 1,270-row "Question-Answer-Context" ground truth set by two domain experts. The reliability of the dataset was validated with a Cohen’s Kappa coefficient of 0.85. Evaluation was conducted using Google Gemini as an LLM-as-a-Judge on a structured 1–5 Likert scale to measure relevance and faithfulness. Experimental findings indicate that the RAG architecture significantly improves reliability by reducing the hallucination rate in the Mistral model from 60% to 35%. The Gemma-7B model emerged as the most successful system, achieving a response time of ~2.4 seconds and an F1 score of 0.28. In conclusion, optimized local RAG architectures provide a robust, privacy-preserving alternative for Turkish educational technologies.

Keywords

• Retrieval-Augmented Generation
• Large Language Models
• Natural Language Processing
• Hallucination
• Educational Technologies
• Local LLM

References

1. [1] Gao, Y. et al. 2023. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv preprint arXiv:2312.10997.
2. [2] Tonmoy, S. M. et al. 2024. A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models. arXiv preprint arXiv:2401.01313.
3. [3] Liu, N. F. et al. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics (TACL), 12, 157-173.
4. [4] Lewis, P. et al. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (NeurIPS), 33, 9459-9474.
5. [5] Ji, Z. et al. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1-38.
6. [6] Kaya, Y. B. and Tantuğ, A. C. 2024. BERT2D: Two Dimensional Positional Embeddings for Efficient Turkish NLP. IEEE Access, 12, 77429-77441.
7. [7] Schweter, S. 2020. BERTurk- BERT models for Turkish. Zenodo.
8. [8] Reimers, N. and Gurevych, I. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP).

9. [9] Chen, J., Lin, H., Han, X. and Sun, L. 2024. Benchmarking Large Language Models in Retrieval-Augmented Generation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16), 17754-17762.
10. [10] Grattafiori, A. et al. 2024. The Llama3 Herd of Models (Version 3). arXiv preprint arXiv:2407.21783.
11. [11] Jiang, A. Q. et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825.
12. [12] Gemma Team. 2024. Gemma: Open Models Based on Gemini Research and Technology. arXiv preprint arXiv:2403.08295.
13. [13] Radford, A. et al. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. International Conference on Machine Learning (ICML).
14. [14] Malkov, Y. A. and Yashunin, D. A. 2020. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4), 824-836.
15. [15] Abdin, M. et al. 2024. Phi3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv preprint arXiv:2404.14219.
16. [16] Zheng, L. et al. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems (NeurIPS), 36.
17. [17] Es, S., James, J., Espinosa Anke, L. and Schockaert, S. 2024. RAGAs: Automated Evaluation of Retrieval Augmented Generation. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 150-158.
18. [18] Karpukhin, V. et al. 2020. Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 6769-6781.
19. [19] Wang, W. et al. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. Advances in Neural Information Processing Systems (NeurIPS), 33, 5776-5788.
20. [20] Guu, K. et al. 2020. Retrieval Augmented Language Model Pre-training. International Conference on Machine Learning (ICML), 3929-3938.
21. [21] Kasneci, E. et al. 2023. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274.