Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives

Abdulkadir Kara; Serkan Yıldırım

doi:10.35377/saucis...1835608

Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives

Abstract

This study examines the performance of large language models (LLMs) in Turkish short-answer assessments within the measurement and evaluation theory framework. The GPT, Gemini, Gemma, and LLaMA models were evaluated under zero-shot and one-shot conditions with rubric support. The results show that LLMs have high internal consistency, but decision reliability can vary depending on prompt format and example sensitivity. Formulating rubrics with clear and concrete performance indicators increases model-human alignment and assessment fairness. Furthermore, error direction analyses revealed that models can exhibit systematic low-scoring tendencies. The results indicate that LLMs can support teachers in formative assessment with properly structured rubrics, but ethical oversight and pedagogical responsibility remain indispensable in final decisions.

Keywords

Ethical Statement

As the research utilises publicly available datasets, it is exempt from ethical approval.

Thanks

We thank Çınar et al. (2020) for sharing the physics dataset as an open data source.

References

A. Çınar, E. Ince, M. Gezer, and O. Yılmaz, “Machine learning algorithm for grading open-ended physics questions in Turkish,” Education and Information Technologies, vol. 25, no. 5, pp. 3821–3844, 2020.
N. I. Kurbanoğlu and M. Olcaytürk, “Investigation of the exam question types attitude scale for secondary school students: Development, validity, and reliability,” Sakarya University Journal of Education, vol. 13, no. 2, pp. 191–206, 2023.
K. Noyes, R. L. McKay, M. Neumann, K. C. Haudek, and M. M. Cooper, “Developing computer resources to automate analysis of students’ explanations of London dispersion forces,” Journal of Chemical Education, vol. 97, no. 11, pp. 3923–3936, 2020.
X. Zhu, H. Wu, and L. Zhang, “Automatic short-answer grading via BERT-based deep neural networks,” IEEE Transactions on Learning Technologies, vol. 15, no. 3, pp. 364–375, 2022.
M. Ariely, T. Nazaretsky, and G. Alexandron, “Personalized automated formative feedback can support students in generating causal explanations in biology,” in Proc. 16th Int. Conf. Learning Sciences (ICLS 2022), 2022, pp. 953–956.
L. Morjaria et al., “Examining the efficacy of ChatGPT in marking short-answer assessments in an undergraduate medical program,” International Medical Education, vol. 3, no. 1, pp. 32–43, 2024.
M. Uto and Y. Uchida, “Automated short-answer grading using deep neural networks and item response theory,” in Artificial Intelligence in Education, AIED 2020, pp. 334–339.
S. Nath, B. Parsaeifard, and E. Werlen, “Automated short answer grading using BERT on German datasets,” in EARLI 2023, 2023.

S. Gombert et al., “Coding energy knowledge in constructed responses with explainable NLP models,” Journal of Computer Assisted Learning, vol. 39, no. 3, pp. 767–786, 2023.
M. Abdul-Salam, M. A. El-Fatah, and N. F. Hassan, “Automatic grading for Arabic short answer questions using optimized deep learning model,” PLOS One, vol. 17, no. 8, p. e0272269, 2022.
W. Westera, M. Dascalu, H. Kurvers, S. Ruseti, and S. Trausan-Matu, “Automated essay scoring in applied games: Reducing the teacher bandwidth problem in online training,” Computers & Education, vol. 123, pp. 212–224, 2018.
O. Henkel, L. Hills, B. Roberts, and J. McGrane, “Can LLMs grade short-answer reading comprehension questions: An empirical study with a novel dataset,” arXiv:2310.18373, 2023.
J. Burstein et al., “Computer analysis of essay content for automated score prediction: A prototype automated scoring system for GMAT analytical writing assessment essays,” ETS Research Report Series, vol. 1998, no. 1, pp. i–67, 1998.
T. K. Landauer, “Latent semantic analysis: A theory of the psychology of language and mind,” Discourse Processes, vol. 27, pp. 303–310, 1999.
C. Leacock and M. Chodorow, “C-rater: Automated scoring of short-answer questions,” Computers and the Humanities, vol. 37, pp. 389–405, 2003.
S. Burrows, I. Gurevych, and B. Stein, “The eras and trends of automatic short answer grading,” International Journal of Artificial Intelligence in Education, vol. 25, pp. 60–117, 2015.
H. A. Ghavidel, A. Zouaq, and M. C. Desmarais, “Using BERT and XLNET for the automatic short answer grading task,” in Proc. CSEDU, 2020, pp. 58–67.
G. Mardini et al., “A deep-learning-based grading system (ASAG) for reading comprehension assessment by using aphorisms as open-answer questions,” Education and Information Technologies, vol. 29, no. 4, pp. 4565–4590, 2024.
J. Sawatzki, T. Schlippe, and M. Benner-Wickner, “Deep learning techniques for automatic short answer grading,” in Proc. Int. Conf. Artificial Intelligence in Education Technology, Singapore, 2021, pp. 65–75.
F. Sun, N. Li, K. Wang, and L. Goette, “Large language models are overconfident and amplify human bias,” arXiv:2505.02151, 2025.
M. Weber and M. Reichardt, “Evaluation is all you need: Prompting generative large language models for annotation tasks in the social sciences,” arXiv:2401.00284, 2023.
T. Brown et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1877–1901.
J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,” arXiv:2201.11903, 2023.
N. Pangakis, S. Wolken, and N. Fasching, “Automated annotation with generative AI requires validation,” arXiv:2306.00176, 2023.
R. Thalken, E. H. Stiglitz, D. Mimno, and M. Wilkens, “Modeling legal reasoning: LM annotation at the edge of human agreement,” arXiv:2310.18440, 2023.
E. Ollion, R. Shen, A. Macanovic, and A. Chatelain, “ChatGPT for text annotation? Mind the hype!,” OSF Preprints, 2023.
A. Spirling, “Why open-source generative AI models are an ethical way forward for science,” Nature, vol. 616, p. 413, 2023.
M. T. Kane, “Validation,” in Educational Measurement, 4th ed., R. L. Brennan, Ed. ACE/Praeger, 2006, pp. 17–64.
S. Messick, “Validity,” in Educational Measurement, 3rd ed., R. L. Linn, Ed. Macmillan, 1989, pp. 13–103.
D. M. Williamson, X. Xi, and F. J. Breyer, “A framework for evaluation and use of automated scoring,” Educational Measurement: Issues and Practice, vol. 31, no. 1, pp. 2–13, 2012.
S. Y. Yoon, “Short answer grading using one-shot prompting and text similarity scoring model,” arXiv:2305.18638, 2023.
C. Senanayake and D. Asanka, “Rubric-based automated short answer scoring using large language models (LLMs),” pp. 1–6, Apr. 2024.
J. Rodriguez et al., “Cross-context adaptability of GPT-4 for automated grading across diverse disciplinary scenarios,” in Proc. Educational Data Mining, 2024.
H. Li, Y. Chu, K. Yang, Y. Copur-Gencturk, and J. Tang, “LLM-based automated grading with human-in-the-Loop,” arXiv:2504.05239, 2025.
Z. Wang and C. Ormerod, “Generative language models with retrieval augmented generation for automated short answer scoring,” arXiv:2408.03811, 2024.
M. Fateen, B. Wang, and T. Suenori Mine, “Beyond scores: A modular RAG-based system for automatic short answer scoring with feedback,” arXiv:2409.20042, 2024.
S. Haldar, S. Sengupta, and A. K. Das, “RUBRA: An agentic AI system for automatic short answer grading using LLMs and RAG,” pp. 1–6, Jul. 2025, doi: 10.1109/ciacon65473.2025.11189329.
A. Kundu and D. Barbosa, “Are large language models good essay graders?,” arXiv:2409.13120, 2024.
G. Kortemeyer, “Performance of the pre-trained large language model GPT-4 on automated short answer grading,” Discover Artificial Intelligence, vol. 4, p. 47, 2024.
R. Gao et al., “Towards scalable automated grading: Leveraging large language models for conceptual question evaluation in engineering,” arXiv:2411.03659, 2024.
P. C. Mendonça, F. Quintal, and F. Mendonça, “Evaluating LLMs for automated scoring in formative assessments,” Applied Sciences, vol. 15, no. 5, p. 2787, Mar. 2025, doi: 10.3390/app15052787.
C. Grevisse, “LLM-based automatic short answer grading in undergraduate medical education,” BMC Medical Education, vol. 24, no. 1, p. 1060, 2024.
W. Li and H. Liu, “Applying large language models for automated essay scoring for non-native Japanese,” Humanities and Social Sciences Communications, vol. 11, pp. 1–15, 2024.
X. Wu et al., “Unveiling scoring processes: Dissecting the differences between LLMs and human graders in automatic scoring,” Technology, Knowledge and Learning, pp. 1–16, 2025.
P. Lai et al., “SAS-Bench: A fine-grained benchmark for evaluating short answer scoring with large language models,” arXiv:2505.07247, 2025.
J. Cohen, “Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit,” Psychological Bulletin, vol. 70, no. 4, pp. 213–220, 1968.
R. Neumann and P. Graeff, “Quantitative approaches to comparative analyses: Data properties and their implications for theory, measurement and modeling,” European Political Science, vol. 14, pp. 385–393, 2015.
T. Brown et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1877–1901.
T. G. Dietterich, “An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization,” Machine Learning, vol. 40, pp. 139–157, 2000.
J. Dodge et al., “Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping,” arXiv:2002.06305, 2020.
C. Yeung et al., “A zero-shot LLM framework for automatic assignment grading in higher education,” arXiv:2501.14305, 2025.
A. A. Khan, O. Chaudhari, and R. Chandra, “A review of ensemble learning and data augmentation models for class imbalanced problems,” Expert Systems with Applications, vol. 244, p. 122778, 2024.
B. Pecher, I. Srba, M. Bielikova, and J. Vanschoren, “Automatic combination of sample selection strategies for few-shot learning,” arXiv:2402.03038, 2024.
L. Yoshida, “The impact of example selection in few-shot prompting on automated essay scoring using GPT models,” in International Conference on Artificial Intelligence in Education, 2024, pp. 61–73.
G. Meyer, P. Breuer, and J. Fürst, “ASAG2024: A combined benchmark for short answer grading,” in Proc. 2024 ACM Virtual Global Computing Education Conf. V.2, 2024, pp. 322–323.
A. Condor, “Automatic grading and feedback for students’ short written responses,” Doctoral dissertation, University of California, Berkeley, 2024.
N. Doewes, A. Kurdhi, and A. Saxena, “Evaluating quadratic weighted kappa as the standard performance metric for automated essay scoring,” in Proc. 16th Int. Conf. Educational Data Mining (EDM 2023), pp. 103–113, 2023.
A. Jonsson and G. Svingby, “The use of scoring rubrics: Reliability, validity and educational consequences,” Educational Research Review, vol. 2, pp. 130–144, 2007.
C. Xavier et al., “Empowering instructors with AI: Evaluating the impact of an AI-driven feedback tool in learning analytics,” IEEE Transactions on Learning Technologies, 2025.
I. Chamieh, T. Zesch, and K. Giebermann, “LLMs in short answer scoring: Limitations and promise of zero-shot and few-shot approaches,” in Proc. 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), 2024, pp. 309–315.
L. H. Chang and F. Ginter, “Automatic short answer grading for Finnish with ChatGPT,” in Proc. AAAI Conf. Artificial Intelligence, vol. 38, no. 21, 2024, pp. 23173–23181.
M. Mynlieff, A. L. Manogaran, M. St. Maurice, and T. J. Eddinger, “Writing assignments with a metacognitive component enhance learning in a large introductory biology course,” CBE—Life Sciences Education, vol. 13, no. 2, pp. 311–321, 2014.

Details

Primary Language

English

Subjects

Computing Applications in Social Sciences and Education, Natural Language Processing

Journal Section

Research Article

Authors

Abdulkadir Kara ^*
0000-0003-3255-1408
Türkiye

Serkan Yıldırım
0000-0002-8277-5963
Türkiye

Early Pub Date

June 25, 2026

Publication Date

June 30, 2026

Submission Date

December 3, 2025

Acceptance Date

February 8, 2026

Published in Issue

Year 2026 Volume: 9 Number: 3

DOI

https://doi.org/10.35377/saucis...1835608

IZ

https://izlik.org/JA79TM88EC

Cite

RIS / Bibtex

APA

Kara, A., & Yıldırım, S. (2026). Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives. Sakarya University Journal of Computer and Information Sciences, 9(3), 980-994. https://doi.org/10.35377/saucis...1835608

AMA

1.Kara A, Yıldırım S. Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives. SAUCIS. 2026;9(3):980-994. doi:10.35377/saucis.1835608

Chicago

Kara, Abdulkadir, and Serkan Yıldırım. 2026. “Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives”. Sakarya University Journal of Computer and Information Sciences 9 (3): 980-94. https://doi.org/10.35377/saucis. 1835608.

EndNote

Kara A, Yıldırım S (June 1, 2026) Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives. Sakarya University Journal of Computer and Information Sciences 9 3 980–994.

IEEE

[1]A. Kara and S. Yıldırım, “Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives”, SAUCIS, vol. 9, no. 3, pp. 980–994, June 2026, doi: 10.35377/saucis...1835608.

ISNAD

Kara, Abdulkadir - Yıldırım, Serkan. “Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives”. Sakarya University Journal of Computer and Information Sciences 9/3 (June 1, 2026): 980-994. https://doi.org/10.35377/saucis. 1835608.

JAMA

1.Kara A, Yıldırım S. Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives. SAUCIS. 2026;9:980–994.

MLA

Kara, Abdulkadir, and Serkan Yıldırım. “Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives”. Sakarya University Journal of Computer and Information Sciences, vol. 9, no. 3, June 2026, pp. 980-94, doi:10.35377/saucis. 1835608.

Vancouver

1.Abdulkadir Kara, Serkan Yıldırım. Evaluating Large Language Models in Turkish Short Answer Scoring: Validity, Reliability, and Fairness Perspectives. SAUCIS. 2026 Jun. 1;9(3):980-94. doi:10.35377/saucis. 1835608