Generative AI in K12: Analytics From Early Adoption

Year 2024, Volume: 15 Issue: Special Issue, 361 - 377, 30.12.2024

Brad Bolender , Sara Vispoel , Geoff Converse , Nick Koprowicz , Dan Song , Sarah Osaro

https://doi.org/10.21031/epod.1539710

Abstract

The integration of generative AI in K12 education and assessment development holds the potential to revolutionize instructional practices, assessment development, and content alignment. This article presents analytical insights and findings from early adoption studies utilizing AI-powered tools developed by Finetune—Generate and Catalog. Generate enhances the efficiency of assessment item development through customized natural language generation, producing high-quality, psychometrically valid items. Catalog intelligently tags and aligns educational content to various standards and frameworks, improving precision and reducing subjectivity. Through three comprehensive case studies, we explore the practical applications, benefits, and lessons learned from employing these AI systems in real-world educational settings. The purpose of this series of studies was to investigate the ways generative AI is currently being used in practical applications in test development to improve processes and products. The studies demonstrate significant reductions in time and costs, enhanced accuracy, and consistency in content alignment, and improved quality of educational and assessment materials. The findings underscore the substantial benefits and critical importance of customized AI systems, rigorous training for both AI models and users, and adopting appropriate evaluation metrics. With the use of off-the-shelf generative AI models expanding rapidly, it is vital that the effectiveness of AI systems that are highly customized through collaborations with measurement experts be presented, in order to maximize benefits and uphold the fundamental principles and best practices of test development.

Keywords

Generative AI, Assessment Development, Content Alignment, Educational Measurement

References

• Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® V. 2. The Journal of Technology, Learning and Assessment, 4(3).
• Bennett, R., & Zhang, M. (2015). Validity and Automated Scoring. Technology and Testing, 142–173. Routledge. https://doi.org/10.4324/9781315871493-8
• Bolender, B., Foster, C., & Vispoel, S. (2023). The Criticality of Implementing Principled Design When Using AI Technologies in Test Development. Language Assessment Quarterly, 20(4-5), 512–519. https://doi.org/10.1080/15434303.2023.2288266
• Burrows, S., Gurevych, I., & Stein, B. (2015). The Eras and Trends of Automatic Short Answer Grading. International Journal of Artificial Intelligence in Education, 25(1), 60–117. https://doi.org/10.1007/s40593-014-0026-8
• Gierl, M. J., & Haladyna, T. M. (2013). Automatic Item Generation: Theory and Practice. https://doi.org/10.4324/9780203803912
• Hao, J., Alina, Yaneva, V., Lottridge, S., Matthias von Davier, & Harris, D. J. (2024). Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI. Educational Measurement. https://doi.org/10.1111/emip.12602
• Ho, A. D. (2024). Artificial Intelligence and Educational Measurement: Opportunities and Threats. Journal of Educational and Behavioral Statistics, 0(0). https://doi.org/10.3102/10769986241248771
• Kaldaras, L., Akaeze, H. O., & Reckase, M. D. (2024). Developing Valid assessments in the Era of Generative Artificial Intelligence. Frontiers in Education (Vol. 9, p. 1399377). https://doi.org/10.3389/feduc.2024.1399377
• Khan, S., Rosaler, J., Hamer, J., & Almeida, T. (2021a). Catalog: An educational content tagging system. In Hsiao, I., Sahebi, S., Bouchet, F., Vie, J. (Eds.), Proceedings of the International Conference on Educational Data Mining, 736-740. International Educational Data Mining Society.
• Khan, S., Hamer, J., & Almeida, T. (2021b). Generate: A NLG system for educational content creation. In Hsiao, I., Sahebi, S., Bouchet, F., Vie, J. (Eds.), Proceedings of the International Conference on Educational Data Mining, 741-744. International Educational Data Mining Society.
• Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10(8), 707-710.
• Yan, D., Rupp, A. A., & Foltz, P. W. (2020). Handbook of Automated Scoring. CRC Press.