Rethinking Automated Scoring in the Era of Artificial Intelligence: Emerging Directions for AI-Based Assessment

Abstract

The rapid development of generative artificial intelligence (AI), particularly large language models and multimodal models, is reshaping research on automated scoring in educational assessment. While previous research has often focused on the extent to which AI-generated scores agree with those assigned by human raters, agreement alone may not provide sufficient evidence for the quality or defensibility of an automated scoring system. This editorial argues that next-generation AI-based scoring research should move beyond model-level performance comparisons and examine the broader conditions under which AI-generated scores are produced, evaluated and used. It first discusses key design dimensions of AI-based scoring systems, including prompt design, rubric design, example-based calibration, model selection, scoring strategies and continuous performance monitoring. It then identifies emerging research directions involving evidence-grounded scoring, uncertainty and confidence calibration, process- and reasoning-aware evaluation, multimodal evidence, retrieval-augmented scoring, fairness and human–AI orchestration. Particular attention is given to iterative multi-agent and deliberative scoring, in which AI systems can review, critique and reconsider previous scoring decisions rather than independently producing a final score. The editorial concludes by proposing a broader research agenda in which automated scoring is evaluated not only in terms of agreement or accuracy, but also validity, reliability, fairness, uncertainty, transparency and the conditions under which human oversight should be incorporated.

Keywords

• Artificial intelligence
• large language model
• automated scoring
• prompt design
• fairness
• multi-agent
• educational assessment
• AI-based scoring
• future directions

References

1. Boduroglu, E., & Yigiter, M. S. (2026). Artificial intelligence scoring attitudes: scale development and validation. Education and Information Technologies, 31(3), 701–726. https://doi.org/10.1007/s10639-025-13836-7
2. Boduroğlu, E., Koç, O., & Yiğiter, M. S. (2023). Madde Güçlüklerinin Tahmin Edilmesinde Uzman Görüşleri ve ChatGPT Performansının Karşılaştırılması / Comparison of Expert Opinions and ChatGPT Performance in Predicting Item Difficulties. Disiplinlerarası Eğitim Araştırmaları Dergisi, 7(15), 202–210. https://doi.org/10.57135/jier.1296255
3. Ercikan, K., & McCaffrey, D. F. (2022). Optimizing implementation of Artificial‐intelligence‐based automated scoring: An evidence centered design approach for designing assessments for AI‐based scoring. Journal of Educational Measurement, 59(3), 272–287. https://doi.org/10.1111/jedm.12332
4. Fianu, E., Amankwah-Sarfo, F., Ofori, P., Amoako, J. K., & Sumani, H. (2026). From traditional machine learning models to large language models: A systematic literature review of automated essay scoring. SN Computer Science, 7(5). https://doi.org/10.1007/s42979-026-05028-y
5. Haudek, K. C., & Zhai, X. (2024). Examining the effect of assessment construct characteristics on machine learning scoring of scientific argumentation. International Journal of Artificial Intelligence in Education, 34(4), 1482–1509. https://doi.org/10.1007/s40593-023-00385-8
6. Humphry, S. M., & Heldsinger, S. A. (2014). Common structural design features of rubrics may represent a threat to validity. Educational Researcher (Washington, D.C.: 1972), 43(5), 253–263. https://doi.org/10.3102/0013189x14542154
7. Jang, J., Moon, A., Jung, M., Kim, Y., & Lee, S. J. (2025). LLM agents at the roundtable: A multi-perspective and dialectical reasoning framework for essay scoring. Findings of the Association for Computational Linguistics: EMNLP 2025, 19674–19687.
8. Jung, J. Y., Tyack, L., & von Davier, M. (2024). Combining machine translation and automated scoring in international large-scale assessments. Large-Scale Assessments in Education, 12(1). https://doi.org/10.1186/s40536-024-00199-7

9. Kim, Y. (2025). Automated essay scoring with GPT ‐4 for a local placement test: Investigating prompting strategies, intra‐rater reliability, and alignment with human scores. TESOL Quarterly, 59(S1). https://doi.org/10.1002/tesq.3405
10. Mughal, N., Imran, A. S., Daudpota, S. M., Kastrati, Z., & Noor, W. (2026). Exploring potential of large language models for automated essay scoring in education. Discover Artificial Intelligence, 6(1). https://doi.org/10.1007/s44163-026-01002-y
11. Pack, A., Barrett, A., & Escalante, J. (2024). Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability. Computers and Education: Artificial Intelligence, 6(100234), 100234. https://doi.org/10.1016/j.caeai.2024.100234
12. Ramesh, D., & Sanampudi, S. K. (2022). An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review, 55(3), 2495–2527. https://doi.org/10.1007/s10462-021-10068-2
13. Vangelova, A., & Gancheva, V. (2026). AI-based automated scoring layer using large language models and semantic analysis. Applied Sciences (Basel, Switzerland), 16(7), 3537. https://doi.org/10.3390/app16073537
14. Voss, E. (2025). Comparison of traditional machine learning and neural network approaches for automated scoring of second language English essays. Language Testing, 42(4), 369–396. https://doi.org/10.1177/02655322251348959
15. Wang, Y., Ding, Z., Wu, X., Sun, S., Liu, N., & Zhai, X. (2026). AutoSCORE: Enhancing automated scoring with multi-agent large language models via Structured COmponent REcognition. Proceedings of the AAAI Conference on Artificial Intelligence, 40(48), 40898–40906. https://doi.org/10.1609/aaai.v40i48.42123
16. Xue, M., Liu, Y., Xiao, X., & Wilson, M. (2025). Automatic prompt engineering for automatic scoring. Journal of Educational Measurement, 62(4), 559–587. https://doi.org/10.1111/jedm.70002
17. Yandi, A. (2026). Operating characteristics of agreement metrics in AI-based scoring: a Monte Carlo simulation. Frontiers in Psychology, 17(1705653), 1705653. https://doi.org/10.3389/fpsyg.2026.1705653
18. Yiğiter, M., & Boduroğlu, E. (2025). Examining the performance of Artificial Intelligence in scoring students’ handwritten responses to open-ended items. TED EĞİTİM VE BİLİM, 50, 1–18. https://doi.org/10.15390/eb.2025.14119 (Original work published as Öğrencilerin El Yazısıyla Yanıtladığı Açık Uçlu Maddelerin Puanlanmasında Yapay Zekâ Performansının İncelenmesi)
19. Yoshida, L. (2025). Do we need a detailed rubric for automated essay scoring using large language models? In Lecture Notes in Computer Science (pp. 60–67). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-98465-5_8
20. Zesch, T., Horbach, A., & Zehner, F. (2023). To score or not to score: Factors influencing performance and feasibility of automatic content scoring of text responses. Educational Measurement Issues and Practice, 42(1), 44–58. https://doi.org/10.1111/emip.12544
21. Zhang, D.-W., Boey, M., Tan, Y. Y., & Jia, A. H. S. (2024). Evaluating large language models for criterion-based grading from agreement to consistency. Npj Science of Learning, 9(1), 79. https://doi.org/10.1038/s41539-024-00291-1