A Semantic and Lexical Analysis of LLM-Generated Responses in Comparison with Stack Overflow Answers

Kerem Can Çelepkolu; Yasin Ortakcı; Kürşat Mustafa Karaoğlan

doi:10.35377/saucis...1865546

A Semantic and Lexical Analysis of LLM-Generated Responses in Comparison with Stack Overflow Answers

Abstract

Large Language Models (LLMs) have become indispensable tools in programming education and professional software development. However, their output quality remains poorly characterized, particularly in terms of the semantic and linguistic quality of the explanatory text. Traditional evaluation metrics such as ROUGE primarily rely on lexical overlap and therefore only partially reflect the semantic adequacy of code explanations. This study proposes a multi-metric evaluation framework integrating semantic embeddings and lexical metrics to assess the quality of explanations generated by LLMs. We conducted experiments on 442 question-answer pairs produced by human experts regarding Python programming language from Stack Overflow, comparing them with GPT-4o-mini outputs. Three embedding models (MiniLM-L6, MPNet-Base, Paraphrase-MiniLM) were employed to compute the measure of semantic similarity by using cosine similarity, and ROUGE metrics (ROUGE-1, ROUGE-2, ROUGE-L) were used for lexical analysis. Results reveal a high semantic alignment (cosine similarity: 0.65–0.75) but substantially lower lexical overlap (ROUGE:0.10–0.32), indicating that LLMs produce semantically sound yet lexically distinct explanations. Pearson correlation analysis reveals weak to moderate correlations (r=0.29–0.47) between semantic and lexical metrics, suggesting that traditional metrics alone are insufficient for a comprehensive evaluation. Statistical testing (paired t-tests, p < 0.001) shows significant differences among embedding models, with MPNet-Base achieving the highest semantic consistency. Our findings underscore the critical need for multidimensional evaluation frameworks in assessing LLM-generated explanations, particularly in technical domains where semantic accuracy is crucial.

Keywords

Ethical Statement

It is declared that during the preparation process of this study, scientific and ethical principles were followed and all the studies benefited from are stated in the bibliography.

References

M. Chen et al., ”Evaluating large language models trained on code,” 2021, arXiv:2107.03374.
P. Vaithilingam, T. Zhang, and E. L. Glassman, ”Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in CHI Conf. Human Factors Comput. Syst. Extended Abstracts, 2022, pp. 1–7, doi: 10.1145/3491101.3519665.
N. Nguyen and S. Nadi, ”An empirical evaluation of GitHub copilot’s code suggestions,” in Proc. 19th Int. Conf. Mining Softw. Repositories (MSR), 2022, pp. 1–5, doi: 10.1145/3524842.3528470.
S. Barke, M. B. James, and N. Polikarpova, ”Grounded copilot: How programmers interact with code-generating models” in Proc. ACM Program. Lang., vol. 7, pp. 85–111, 2023.
OpenAI, ”GPT-4 technical report”, 2023, arXiv:2303.08774.
R. Anil et al., PaLM 2 technical report, 2023, arXiv:2305.10403.
S. Kabir, D. N. Udo-Imeh, B. Kou, T. Zhang, ”Is Stack Overflow obsolete? An empirical study of the characteristics of ChatGPT answers to Stack Overflow questions”, 2023, arXiv:2308.02312.
J. Prather et al., ”The robots are here: Navigating the generative AI revolution in computing education,” in Proc. 2023 Working Group Reports Innov. Technol. Comput. Sci. Educ., 2023, pp. 108–159.

H. Tian et al., ”Is ChatGPT the ultimate programming assistant–how far is it?,” 2023, arXiv:2304.11938.
Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, C. Zhu, ”G-eval: NLG evaluation using GPT-4 with better human alignment’,’ 2023, arXiv:2303.16634.
Y. Bang et al., ”A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity,” 2023, arXiv:2302.04023.
Z. Ji et al., ”Survey of hallucination in natural language generation,” ACM Comput. Surv, vol. 55, pp. 1–38, 2023.
Y. Zhang et al., ”Siren’s song in the AI ocean: A survey on hallucination in large language models,” 2023, arXiv:2309.01219.
L. Mamykina, B. Manoim, M. Mittal, G. Hripcsak, and B. Hartmann, ”Design lessons from the fastest Q&A site in the west” in Proc. SIGCHI Conf. Human Factors Comput. Syst., 2011, pp. 2857–2866.
S. Gehrmann, E. Clark, and T. Sellam, ”Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text” J. Artif. Intell. Res., vol. 77, pp. 103–166, 2023.
M. Zhong, Y. Liu, D. Yin, Y. Miao, P. Liu, and D. Hakkani-Tur, ”Towards a unified multi-dimensional evaluator for text generation,” 2022, arXiv:2210.07197.
K. Papineni, S. Roukos, T. Ward, and W. J. Zhu, ”Bleu: A method for automatic evaluation of machine translation,” in Proc. 40th Annu. Meeting Assoc. Comput. Linguistics, 2002, pp. 311–318.
C. Y. Lin, ”Rouge: A package for automatic evaluation of summaries,” Text Summarization Branches Out, 2004, pp. 74–81.
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, ”BERTScore: Evaluating text generation with BERT,” 2019, arXiv:1904.09675
A. Celikyilmaz, E. Clark, and J. Gao, ”Evaluation of text generation: A survey,” 2020, arXiv:2006.14799.
N. Reimers and I. Gurevych, ”Sentence-BERT: Sentence embeddings using siamese BERT-networks,” in Proc. 2019 Conf. Empirical Methods Natural Language Processing, 2019, pp. 3982–3992.
S. Haque, Z. Eberhart, A. Bansal, and C. McMillan, ”Semantic similarity metrics for evaluating source code summarization,” in Proc. 30th IEEE/ACM Int. Conf. Program Comprehension (ICPC), 2022, pp. 36–47, doi: 10.1145/3524610.3527909.
W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, ”MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,” Adv. Neural Inf. Process. Syst., vol. 33, pp. 5776–5788, 2020.
K. Song, X. Tan, T. Qin, J. Lu, and T. Y. Liu, ”MPNet: Masked and permuted pre-training for language understanding,” Adv. Neural Inf. Process. Syst., vol. 33, pp. 16857–16867, 2020.
J. Austin et al., ”Program synthesis with large language models,” 2021, arXiv:2108.07732.
J. Liu, C. S. Xia, Y. Wang, and L. Zhang, ”Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation,” 2023, arXiv:2305.01210.
N. Thakur, N. Reimers, A. R¨uckl´e, A. Srivastava, and I. Gurevych, ”BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models,” 2021, arXiv:2104.08663.
N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, ”MTEB: Massive text embedding benchmark,” 2022, arXiv:2210.07316.
L. Ouyang et al., ”Training language models to follow instructions with human feedback,” Adv. Neural Inf. Process. Syst., vol. 35, pp. 27730–27744, 2022.
M. F. Baer and R. S. Purves, ”Identifying landscape relevant natural language using actively crowdsourced landscape descriptions and sentence-transformers,” KI - K¨unstliche Intelligenz, vol. 37, pp. 55–67, 2023, doi: 10.1007/s13218-022-00793-3.
E. Elahi, J. Morato, and A. Iglesias, ”Improving web readability using video Content: A relevance-based approach,” Appl. Sci., vol. 14, no. 23, Art. no. 11055, 2024, doi: 10.3390/app142311055.
J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ, USA: Lawrence Erlbaum Associates, 1988.

Details

Primary Language

English

Subjects

Natural Language Processing

Journal Section

Research Article

Authors

Kerem Can Çelepkolu
0009-0000-4297-5437
Türkiye

Yasin Ortakcı ^*
0000-0002-0683-2049
Türkiye

Kürşat Mustafa Karaoğlan
0000-0001-9830-7622
Türkiye

Early Pub Date

October 27, 2026

Publication Date

-

Submission Date

February 21, 2026

Acceptance Date

August 22, 2026

Published in Issue

Year 2026 Number: Advanced Online Publication

DOI

https://doi.org/10.35377/saucis...1865546

IZ

https://izlik.org/JA69ZG25HE

Cite

RIS / Bibtex

APA

Çelepkolu, K. C., Ortakcı, Y., & Karaoğlan, K. M. (2026). A Semantic and Lexical Analysis of LLM-Generated Responses in Comparison with Stack Overflow Answers. Sakarya University Journal of Computer and Information Sciences, Advanced Online Publication, 1485-1493. https://doi.org/10.35377/saucis...1865546

AMA

1.Çelepkolu KC, Ortakcı Y, Karaoğlan KM. A Semantic and Lexical Analysis of LLM-Generated Responses in Comparison with Stack Overflow Answers. SAUCIS. 2026;(Advanced Online Publication):1485-1493. doi:10.35377/saucis.1865546

Chicago

Çelepkolu, Kerem Can, Yasin Ortakcı, and Kürşat Mustafa Karaoğlan. 2026. “A Semantic and Lexical Analysis of LLM-Generated Responses in Comparison With Stack Overflow Answers”. Sakarya University Journal of Computer and Information Sciences, no. Advanced Online Publication: 1485-93. https://doi.org/10.35377/saucis. 1865546.

EndNote

Çelepkolu KC, Ortakcı Y, Karaoğlan KM (October 1, 2026) A Semantic and Lexical Analysis of LLM-Generated Responses in Comparison with Stack Overflow Answers. Sakarya University Journal of Computer and Information Sciences Advanced Online Publication 1485–1493.

IEEE

[1]K. C. Çelepkolu, Y. Ortakcı, and K. M. Karaoğlan, “A Semantic and Lexical Analysis of LLM-Generated Responses in Comparison with Stack Overflow Answers”, SAUCIS, no. Advanced Online Publication, pp. 1485–1493, Oct. 2026, doi: 10.35377/saucis...1865546.

ISNAD

Çelepkolu, Kerem Can - Ortakcı, Yasin - Karaoğlan, Kürşat Mustafa. “A Semantic and Lexical Analysis of LLM-Generated Responses in Comparison With Stack Overflow Answers”. Sakarya University Journal of Computer and Information Sciences. Advanced Online Publication (October 1, 2026): 1485-1493. https://doi.org/10.35377/saucis. 1865546.

JAMA

1.Çelepkolu KC, Ortakcı Y, Karaoğlan KM. A Semantic and Lexical Analysis of LLM-Generated Responses in Comparison with Stack Overflow Answers. SAUCIS. 2026;:1485–1493.

MLA

Çelepkolu, Kerem Can, et al. “A Semantic and Lexical Analysis of LLM-Generated Responses in Comparison With Stack Overflow Answers”. Sakarya University Journal of Computer and Information Sciences, no. Advanced Online Publication, Oct. 2026, pp. 1485-93, doi:10.35377/saucis. 1865546.

Vancouver

1.Kerem Can Çelepkolu, Yasin Ortakcı, Kürşat Mustafa Karaoğlan. A Semantic and Lexical Analysis of LLM-Generated Responses in Comparison with Stack Overflow Answers. SAUCIS. 2026 Oct. 1;(Advanced Online Publication):1485-93. doi:10.35377/saucis. 1865546