Evaluating Large Language Models with a Unified Prompt-Based Pipeline for Turkish News Summarization and Tag Extraction

Burcu Koçak; Kazım Yıldız

doi:10.35377/saucis...1864801

Evaluating Large Language Models with a Unified Prompt-Based Pipeline for Turkish News Summarization and Tag Extraction

Abstract

As access to information grows ever faster, the acceleration of news flow demands the automatic and accurate analysis of texts. This study aims to measure the multitasking performance of summarizing and tag extraction tasks in Turkish news texts using a prompt-based unified pipeline with a small dataset, employing different prompt techniques. Summarization and tag extraction tasks were performed comparatively using zero-shot and few-shot learning approaches with commercial models (GPT-40 and Claude 4.5 Sonnet) via official API providers and open-source models (Qwen3-32B and Llama 3.3-70B-Instruct) on a local server using a vLLM infrastructure. Experiments were conducted with deterministic parameters and standardized system prompts. Experimental results show that the Claude 4.5 Sonnet model exhibited the highest performance in the 3-shot scenario with a BERTScore of 0.676 and an LLM-as-a-judge score of 7.67. Furthermore, it has been observed that switching to a few-shot strategy resulted in up to a 35% increase in tagging success. The Llama 3.3 model was found to be competitive with commercial models in the tagging task with an F1 score of 0.287. These results also validate the proposed unified pipeline method in terms of accuracy and semantic similarity using BertScore, F1 score, and the LLM-as-a-judge method, which simulates human judgment. The proposed framework is expected to contribute to the literature by offering a reliable automation solution that increases efficiency with small data requirements in Turkish news processing processes.

Keywords

Ethical Statement

Scientific and ethical principles were followed. No ethical committee approval was required as the study used scraped public news data without human/animal subjects.

Thanks

The authors would like to thank TRT Haber for the news data and the open-source community for providing the Llama and Qwen models. Special thanks to the research supervisor for the technical guidance and support throughout this work.

References

B. Baykara and T. Güngör, “Abstractive text summarization and new large-scale datasets for agglutinative languages Turkish and Hungarian,” Language Resources and Evaluation, vol. 56, no. 3, p. 973–1007, 2022. [Online]. Available: https://doi.org/10.1007/s10579-021-09568-y
M. Akashvarma, G. Yashaswini, E. Keerthana, S. Siddharth, K. Saipooja, and G. Prasanna Kumar, “A comprehensive review of large language models in abstractive summarization of news articles,” in 2024 Asia Pacific Conference on Innovation in Technology (APCIT). IEEE, Conference Proceedings, p. 1–6.
M. Cate, “The role of zero-shot and few-shot learning in enhancing llms for real-world applications,” 2023.
I. Borhan and A. Bajaj, “The effect of prompt types on text summarization performance with large language models,” Journal of Database Management (JDM), vol. 35, no. 1, p. 1–23, 2024.
Z. A. Guven, “Performance comparison of large language models, gpt and gemini on turkish news classification task,” 08 May 2024.
Y. Chang et al., “A survey on evaluation of large language models,” ACM transactions on intelligent systems and technology, vol. 15, no. 3, p. 1–45, 2024.
K. Chang et al., “Efficient prompting methods for large language models: A survey,” arXiv preprint arXiv:2404.01077, 2024.
J. Gu et al., “A survey on llm-as-a-judge,” The Innovation, 2024.

T. Scialom, P.-A. Dray, S. Lamprier, B. Piwowarski, and J. Staiano, “Mlsum: The multilingual summarization corpus,” arXiv preprint arXiv:2004.14900, 2020.
T. Hasan et al., “Xl-sum: Large-scale multilingual abstractive summarization for 44 languages,” arXiv preprint arXiv:2106.13822, 2021.
T. Zhang, F. Ladhak, E. Durmus, P. Liang, K. McKeown, and T. B. Hashimoto, “Benchmarking large language models for news summarization,” Transactions of the Association for Computational Linguistics, vol. 12, p. 39–57, 2024.
R. F. Cekinel, P. Karagoz, and C. Coltekin, “Cross-lingual learning vs. low-resource fine-tuning: A case study with fact-checking in turkish,” arXiv preprint arXiv:2403.00411, 2024.
C. Üyük, D. Rovó, S. Kolli, R. Varol, G. Groh, and D. Dementieva, “Crafting tomorrow’s headlines: Neural news generation and detection in english, turkish, hungarian, and persian,” arXiv preprint arXiv:2408.10724, 2024.
A. Yüksel, A. Köksal, L. K. Şenel, A. Korhonen, and H. Schütze, “Turkishmmlu: Measuring massive multitask language understanding in turkish,” arXiv preprint arXiv:2407.12402, 2024.
A. Kuş and Ç. İ. Acı, “A hybrid approach to automatic text summarization of turkish texts: Integrating extractive methods with llms,” in 2024 Innovations in Intelligent Systems and Applications Conference (ASYU). IEEE, Conference Proceedings, p. 1–6.
N. Z. Kayalı and S. I. Omurca, “Htv-news: A new dataset with high novelty rate for turkish text summarization,” in 2024 9th International Conference on Computer Science and Engineering (UBMK). IEEE, Conference Proceedings, p. 1–6.
G. Uludoğan, Z. Y. Balal, F. Akkurt, M. Türker, O. Güngör, and S. Üsküdarlı, “Turna: A turkish encoder-decoder language model for enhanced understanding and generation,” arXiv preprint arXiv:2401.14373, 2024.
M. Turker, M. E. Ari, and A. Han, “Vbart: The turkish llm,” arXiv preprint arXiv:2403.01308, 2024.
Y. A. Er, I. Kesen, G. G. Şahin, and A. Erdem, “Cetvel: A unified benchmark for evaluating language understanding, generation and cultural capacity of llms for turkish,” Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1052–1085, 2026.
M. A. Bayram, A. A. Fincan, A. S. Gümüş, B. Diri, S. Yıldırım, and Ö. Aytaş, “Setting standards in turkish nlp: Tr-mmlu for large language model evaluation,” arXiv preprint arXiv:2501.00593, 2024.
F. Jin, J. Lu, and J. Zhang, “Unified prompt learning makes pre-trained language models better few-shot learners,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, Conference Proceedings, p. 1–5.
W. M. Aly, T. H. A. Soliman, and A. M. AbdelAziz, “An evaluation of large language models on text summarization tasks using prompt engineering techniques,” arXiv preprint arXiv:2507.05123, 2025.
A. Odabaşı and G. Biricik, “Unraveling the capabilities of language models in news summarization,” arXiv preprint arXiv:2501.18128, 2025.
TRT Haber, “Trt haber web portal,” https://www.trthaber.com/, 2025, accessed: December. 15, 2025.
OpenAI, “Gpt-4o system card,” 2024, (accessed 29 October, 2025). [Online]. Available: https://openai.com/index/gpt-4o-system-card/
E. Glover, “What is gpt-4o?” 2024, (accessed 29 October, 2025). [Online]. Available: https://builtin.com/artificial-intelligence/what-is-gpt4o
R. Islam and O. M. Moushi, “Gpt-4o: The cutting-edge advancement in multimodal llm,” in Intelligent Computing-Proceedings of the Computing Conference. Springer, Conference Proceedings, p. 47–60.
OpenAI, “Hello gpt-4o,” 2024, (accessed 30 October, 2025). [Online]. Available: https://openai.com/index/hello-gpt-4o/
Anthropic, “Claude sonnet 4.5 system card,” Report, 2025, Accessed: 30 October 2025. [Online]. Available: https://assets.anthropic.com/m/12f214efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf
Anthropic, “Introducing claude sonnet 4.5,” 2025, (accessed: 30 October 2025.). [Online]. Available: https://www.anthropic.com/news/claude-sonnet-4-5
Anthropic, “Models overview,” 2025, (accessed: 31 October 2025.). [Online]. Available: https://docs.claude.com/en/docs/about-claude/models/overview#model-comparison-table
G. Ojeda, “Claude sonnet 4.5: Highest-scoring claude model yet on swe-bench,” 2025, (accessed: 31 October 2025.). [Online]. Available: https://caylent.com/blog/claude-sonnet-4-5-highest-scoring-claude-model-yet-on-swe-bench
I. Meta Platforms, “Llama 3.3-70b-instruct,” 2024, Accessed: 31 October 2025. [Online]. Available: https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct
A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, and C. Lv, “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025.
A. Cloud, “Qwen/qwen3-32b,” 2025, Accessed: 4 January 2026. [Online]. Available: https://huggingface.co/Qwen/Qwen3-32B
C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out, 2004, pp. 74–81.
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675, 2019.
K. M. Sujon, R. Hassan, K. Choi, and M. A. Samad, “Accuracy, precision, recall, f1-score, or mcc? empirical evidence from advanced statistics, ml, and xai for evaluating business predictive models,” Journal of Big Data, vol. 12, no. 1, p. 268, 2025.

Details

Primary Language

English

Subjects

Artificial Intelligence (Other)

Journal Section

Research Article

Authors

Burcu Koçak
0009-0006-5428-6659
Türkiye

Kazım Yıldız ^*
0000-0001-6999-1410
Türkiye

Publication Date

September 30, 2026

Submission Date

January 16, 2026

Acceptance Date

March 30, 2026

Published in Issue

Year 2026 Volume: 9 Number: 4

DOI

https://doi.org/10.35377/saucis...1864801

IZ

https://izlik.org/JA58GB38ZJ

Cite

RIS / Bibtex

APA

Koçak, B., & Yıldız, K. (2026). Evaluating Large Language Models with a Unified Prompt-Based Pipeline for Turkish News Summarization and Tag Extraction. Sakarya University Journal of Computer and Information Sciences, 9(4), 1127-1140. https://doi.org/10.35377/saucis...1864801

AMA

1.Koçak B, Yıldız K. Evaluating Large Language Models with a Unified Prompt-Based Pipeline for Turkish News Summarization and Tag Extraction. SAUCIS. 2026;9(4):1127-1140. doi:10.35377/saucis.1864801

Chicago

Koçak, Burcu, and Kazım Yıldız. 2026. “Evaluating Large Language Models With a Unified Prompt-Based Pipeline for Turkish News Summarization and Tag Extraction”. Sakarya University Journal of Computer and Information Sciences 9 (4): 1127-40. https://doi.org/10.35377/saucis. 1864801.

EndNote

Koçak B, Yıldız K (September 1, 2026) Evaluating Large Language Models with a Unified Prompt-Based Pipeline for Turkish News Summarization and Tag Extraction. Sakarya University Journal of Computer and Information Sciences 9 4 1127–1140.

IEEE

[1]B. Koçak and K. Yıldız, “Evaluating Large Language Models with a Unified Prompt-Based Pipeline for Turkish News Summarization and Tag Extraction”, SAUCIS, vol. 9, no. 4, pp. 1127–1140, Sept. 2026, doi: 10.35377/saucis...1864801.

ISNAD

Koçak, Burcu - Yıldız, Kazım. “Evaluating Large Language Models With a Unified Prompt-Based Pipeline for Turkish News Summarization and Tag Extraction”. Sakarya University Journal of Computer and Information Sciences 9/4 (September 1, 2026): 1127-1140. https://doi.org/10.35377/saucis. 1864801.

JAMA

1.Koçak B, Yıldız K. Evaluating Large Language Models with a Unified Prompt-Based Pipeline for Turkish News Summarization and Tag Extraction. SAUCIS. 2026;9:1127–1140.

MLA

Koçak, Burcu, and Kazım Yıldız. “Evaluating Large Language Models With a Unified Prompt-Based Pipeline for Turkish News Summarization and Tag Extraction”. Sakarya University Journal of Computer and Information Sciences, vol. 9, no. 4, Sept. 2026, pp. 1127-40, doi:10.35377/saucis. 1864801.

Vancouver

1.Burcu Koçak, Kazım Yıldız. Evaluating Large Language Models with a Unified Prompt-Based Pipeline for Turkish News Summarization and Tag Extraction. SAUCIS. 2026 Sep. 1;9(4):1127-40. doi:10.35377/saucis. 1864801