Modelling a dataset-defined skill-retention outcome in generative AI-assisted learning: An exploratory machine-learning and explainability study

Abstract

Generative artificial intelligence (GenAI) is increasingly used in higher education, but durable learning cannot be inferred from task completion alone. This study presents an exploratory machine-learning benchmark using a public dataset of 50,000 student-like records that is treated here as synthetic/engineered because its source does not document an empirical sampling frame, institution, country, recruitment process, response rate, or primary-study ethics procedures. The target field, Skill Retention Score, is defined in the source schema on a 0-100 scale as representing skills retained and applied after the semester; however, the source provides no assessment instrument, delayed-assessment interval, reliability estimate, or validity evidence. Accordingly, the variable is analysed as a dataset-defined proxy rather than a validated measure of long-term retention. Ridge Regression, Decision Tree Regression, and Extra Trees Regression were compared in the originally reported 80:20 hold-out analysis. On that split, Extra Trees produced R² = .185, RMSE = 11.983, and MAE = 9.696. Its RMSE is approximately 9.8% lower than the full-sample outcome standard deviation of 13.282, indicating modest predictive gain. The explainability output is permutation importance aggregated to the parent-variable level; it is interpreted as a model-internal, non-directional diagnostic, and SHAP results were not available in the submitted analytical record. The results therefore describe the structure of this benchmark dataset and the behaviour of the modelling pipeline; they do not establish effects of GenAI on real students, directional relationships, or instructional recommendations.

Keywords

• Generative AI
• skill-retention proxy
• synthetic data
• machine learning
• learning analytics

References

1. Abbas, M., Jam, F. A., & Khan, T. I. (2024). Is it harmful or helpful? Examining the causes and consequences of generative AI usage among university students. International Journal of Educational Technology in Higher Education, 21, Article 10. https://doi.org/10.1186/s41239-024-00444-7
2. Baidoo-Anu, D., Asamoah, D., Amoako, I., & Mahama, I. (2024). Exploring student perspectives on generative artificial intelligence in higher education learning. Discover Education, 3, Article 98. https://doi.org/10.1007/s44217-024-00173-z
3. Bittle, K., & El-Gayar, O. (2025). Generative AI and academic integrity in higher education: A systematic review and research agenda. Information, 16(4), Article 296. https://doi.org/10.3390/info16040296
4. Carmi, G. (2025). Learning with generative AI: An empirical study of students in higher education. Education Sciences, 15(12), Article 1696. https://doi.org/10.3390/educsci15121696
5. Gao, Z., Cheah, J.-H., Lim, X.-J., & Luo, X. (2024). Enhancing academic performance of business students using generative AI: An interactive-constructive-active-passive (ICAP) self-determination perspective. The International Journal of Management Education, 22(2), Article 100958. https://doi.org/10.1016/j.ijme.2024.100958
6. Haderlie, T. C., Jr., Sewak, S., Agarwal, S., & Lee, J. (2025). Investigating the effects of generative AI on student satisfaction and self-efficacy. Marketing Education Review. https://doi.org/10.1080/10528008.2025.2576159
7. Henderson, M., Bearman, M., Chung, J., Fawns, T., Buckingham Shum, S., Matthews, K. E., & de Mello Heredia, J. (2025). Comparing generative AI and teacher feedback: Student perceptions of usefulness and trustworthiness. Assessment & Evaluation in Higher Education. https://doi.org/10.1080/02602938.2025.2502582
8. Hon, K. K. L. (2026). Generative AI in higher education: A systematic review of its effects on learning outcomes and academic performance. Journal of Educational Technology Systems, 54(3), 537–560. https://doi.org/10.1177/00472395251400089

9. Jia, W., Pan, L., & Neary, S. (2025). Effect of GenAI dependency on university students’ academic achievement: The mediating role of self-efficacy and moderating role of perceived teacher caring. Behavioral Sciences, 15(10), Article 1348. https://doi.org/10.3390/bs15101348
10. Johnston, H., Wells, R. F., Shanks, E. M., Boey, T., & Parsons, B. N. (2024). Student perspectives on the use of generative artificial intelligence technologies in higher education. International Journal for Educational Integrity, 20, Article 2. https://doi.org/10.1007/s40979-024-00149-4
11. Khosravi, H., Shibani, A., Jovanovic, J., Pardos, Z. A., & Yan, L. (2025). Generative AI and learning analytics: Pushing boundaries, preserving principles. Journal of Learning Analytics, 12(1), 1–11. https://doi.org/10.18608/jla.2025.8961
12. Kim, J., Klopfer, M., Grohs, J. R., Eldardiry, H., Weichert, J., Cox, L. A., II, & Pike, D. (2025). Examining faculty and student perceptions of generative AI in university courses. Innovative Higher Education, 50, 1281–1313. https://doi.org/10.1007/s10755-024-09774-w
13. Lee, D., & Palmer, E. (2025). Prompt engineering in higher education: A systematic review to help inform curricula. International Journal of Educational Technology in Higher Education, 22, Article 7. https://doi.org/10.1186/s41239-025-00503-7
14. McGrath, C., Farazouli, A., & Cerratto-Pargman, T. (2025). Generative AI chatbots in higher education: A review of an emerging research area. Higher Education, 89, 1533–1549. https://doi.org/10.1007/s10734-024-01288-w
15. Misiejuk, K., López-Pernas, S., Kaliisa, R., & Saqr, M. (2025). Mapping the landscape of generative artificial intelligence in learning analytics: A systematic literature review. Journal of Learning Analytics, 12(1), 12–31. https://doi.org/10.18608/jla.2025.8591
16. Pallant, J. L., Blijlevens, J., Campbell, A., & Jopp, R. (2025). Mastering knowledge: The impact of generative AI on student learning outcomes. Studies in Higher Education, 1–22. https://doi.org/10.1080/03075079.2025.2487570
17. Park, J., & Choo, S. (2025). Generative AI prompt engineering for educators: Practical strategies. Journal of Special Education Technology, 40(3). https://doi.org/10.1177/01626434241298954
18. Qian, Y. (2025). Pedagogical applications of generative AI in higher education: A systematic review of the field. TechTrends, 69, 1105–1120. https://doi.org/10.1007/s11528-025-01100-1
19. Qian, Y. (2025). Prompt engineering in education: A systematic review of empirical studies. Journal of Educational Computing Research. https://doi.org/10.1177/07356331251365189
20. Rodríguez-Ortiz, M. Á., Santana-Mancilla, P. C., & Anido-Rifón, L. E. (2025). Machine learning and generative AI in learning analytics for higher education: A systematic review of models, trends, and challenges. Applied Sciences, 15(15), Article 8679. https://doi.org/10.3390/app15158679
21. sisiro, N. (2026). Impact of Ai on Students [Data set]. Kaggle. https://www.kaggle.com/datasets/laveshjadon/ai-impact-on-students
22. Sousa, A. E., & Cardoso, P. (2025). Use of generative AI by higher education students. Electronics, 14(7), Article 1258. https://doi.org/10.3390/electronics14071258
23. Weng, X., & Chiu, T. K. F. (2024). Assessment and learning outcomes for generative AI in higher education: A scoping review. Australasian Journal of Educational Technology, 40(4), 1–16.
24. Weng, X., Xia, Q., Gu, M., Rajaram, K., & Chiu, T. K. F. (2024). Assessment and learning outcomes for generative AI in higher education: A scoping review on current research status and trends. Australasian Journal of Educational Technology, 40(6), 37–55. https://doi.org/10.14742/ajet.9540
25. Yusuf, A., Pervin, N., Román-González, M., & Noor, N. M. (2024). Generative AI and the future of higher education: A threat to academic integrity or reformation? Evidence from multicultural perspectives. International Journal of Educational Technology in Higher Education, 21, Article 21. https://doi.org/10.1186/s41239-024-00453-6
26. Zhidkikh, D., Heilala, V., Van Petegem, C., Dawyndt, P., Järvinen, M., Viitanen, S., De Wever, B., Mesuere, B., Lappalainen, V., Kettunen, L., & Hämäläinen, R. (2024). Reproducing predictive learning analytics in CS1: Toward generalizable and explainable models for enhancing student retention. Journal of Learning Analytics, 11(1), 132–150. https://doi.org/10.18608/jla.2024.7979