Benchmarking and interpreting a tabular foundation model for PISA 2022 mathematical literacy

Abstract

Recent advances in machine learning have expanded the analytical toolkit available for large-scale assessment data; however, many high-performing approaches remain limited in terms of interpretability and their suitability for assessment-informed decision-making. This study used data from 7,250 students in the Türkiye sample of PISA 2022 mathematical literacy to benchmark and interpret the Tabular Prior-Data Fitted Network (TabPFN) for classifying student achievement. For the purpose of model evaluation, mathematical literacy was operationalised as the mean of ten plausible values and then converted into a binary outcome using the PISA Level 2 proficiency benchmark, consistent with the study’s classification-oriented aim. TabPFN was compared with eXtreme Gradient Boosting (XGBoost), Light Gradient-Boosting Machine (LightGBM), Stacking, Random Forest, and Logistic Regression under consistent preprocessing and stratified cross-validation procedures. Beyond predictive performance, feature-importance patterns were examined using Shapley Additive Explanations (SHAP) for TabPFN and permutation importance for the baseline models. The findings showed that TabPFN achieved competitive performance and produced an interpretable predictor profile. Home resources and mathematics self-efficacy emerged as the most influential predictors, followed by curiosity and classroom disciplinary climate. These results suggest that tabular foundation models can support large-scale educational assessment not only by providing accurate classifications, but also by generating interpretable evidence about achievement-related learner and contextual factors. The findings should be interpreted as classification evidence around a meaningful proficiency threshold rather than as a model of continuous mathematics achievement.

Keywords

• Large-scale assessment
• Machine learning
• TabPFN
• Learning analytics
• Mathematical literacy

Thanks

We thank the OECD for making the PISA 2022 database publicly available, which made this research possible.

References

1. Akyüz, G., & Pala, N.M. (2010). PISA 2003 sonuçlarına göre öğrenci ve sınıf özelliklerinin matematik okuryazarlığına ve problem çözme becerilerine etkisi [The effect of student and class characteristics on mathematics literacy and problem solving in PISA 2003]. Elementary Education Online, 9(2), 668–678. https://ilkogretim-online.org/index.php/pub/article/view/761
2. Baker, R.S.J.d., & Inventado, P.S. (2014). Educational data mining and learning analytics. In J.A. Larusson & B. White (Eds.), Learning analytics: From research to practice (pp. 61–75). Springer. https://doi.org/10.1007/978-1-4614-3305-7_4
3. Bakan Kalaycıoğlu, D. (2015). The influence of socioeconomic status, self-efficacy, and anxiety on mathematics achievement in England, Greece, Hong Kong, the Netherlands, Turkey, and the USA. Educational Sciences: Theory & Practice, 15(5), 1391 1401. https://doi.org/10.12738/estp.2015.5.2731
4. Biau, G., & Scornet, E. (2016). A random forest guided tour. TEST, 25(2), 197 227. https://doi.org/10.1007/s11749-016-0481-7
5. Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5 32. https://doi.org/10.1023/A:1010933404324
6. Cheema, J.R., & Kitsantas, A. (2014). Influences of disciplinary classroom climate on high school student self-efficacy and mathematics achievement: A look at gender and racial–ethnic differences. International Journal of Science and Mathematics Education, 12(5), 1261 1279. https://doi.org/10.1007/s10763-013-9454-4
7. Chen, T., & Guestrin, C. (2016, August). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16) (pp. 785 794). Association for Computing Machinery. https://doi.org/10.1145/2939672.2939785
8. Chicco, D., & Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21(1), Article 6. https://doi.org/10.1186/s12864-019-6413-7

9. Hanushek, E.A., & Woessmann, L. (2011). The economics of international differences in educational achievement. In E.A. Hanushek, S. Machin, & L. Woessmann (Eds.), Handbook of the economics of education (Vol. 3, pp. 89–200). Elsevier. https://doi.org/10.1016/B978-0-444-53429-3.00002-8
10. Harris, C.R., Millman, K.J., van der Walt, S.J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N.J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M.H., Brett, M., Haldane, A., Fernández del Río, J., Wiebe, M., Peterson, P., …, Oliphant, T.E. (2020). Array programming with NumPy. Nature, 585(7825), 357–362. https://doi.org/10.1038/s41586-020-2649-2
11. Hollmann, N., Müller, S., Eggensperger, K., & Hutter, F. (2023). TabPFN: A transformer that solves small tabular classification problems in a second. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR 2023). OpenReview. https://doi.org/10.48550/arXiv.2207.01848
12. Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S.B., Schirrmeister, R.T., & Hutter, F. (2025). Accurate predictions on small data with a tabular foundation model. Nature, 637(8045), 319–326. https://doi.org/10.1038/s41586-024-08328-6
13. Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial intelligence in education: Promises and implications for teaching and learning. Center for Curriculum Redesign. https://curriculumredesign.org/our-work/artificial-intelligence-in-education/
14. Hosmer, D.W., Jr., Lemeshow, S., & Sturdivant, R.X. (2013). Applied logistic regression (3rd ed.). Wiley.
15. Hospedales, T., Antoniou, A., Micaelli, P., & Storkey, A. (2022). Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9), 5149–5169. https://doi.org/10.1109/TPAMI.2021.3079209
16. Kamaliyah, K., Zulkardi, Z., & Darmawijoyo, D. (2013). Developing the sixth level of PISA-like mathematics problems for secondary school students. Journal on Mathematics Education, 4(1), 9–28. https://doi.org/10.22342/jme.4.1.562.9-28
17. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). LightGBM: A highly efficient gradient boosting decision tree. In I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 30, pp. 3146 3154). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html
18. Kizilcec, R.F. (2016). How much information? Effects of transparency on trust in an algorithmic interface. In J. Kaye, A. Druin, C. Lampe, D. Morris, & J.P. Hourcade (Eds.), Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16) (pp. 2390–2395). Association for Computing Machinery. https://doi.org/10.1145/2858036.2858402
19. Kizilcec, R.F., & Lee, H. (2022). Algorithmic fairness in education. In W. Holmes & K. Porayska-Pomsta (Eds.), The ethics of artificial intelligence in education (Chap. 10). Routledge. https://doi.org/10.4324/9780429329067-10
20. Liu, X., Yang Hansen, K., Valcke, M., & De Neve, J. (2024). A decade of PISA: Student perceived instructional quality and mathematics achievement across European countries. ZDM – Mathematics Education, 56, 859–891. https://doi.org/10.1007/s11858-024-01630-7
21. Lundberg, S.M., & Lee, S.I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30. https://doi.org/10.48550/arXiv.1705.07874
22. Menard, S. (2002). Applied logistic regression analysis (2nd ed.). Sage.
23. Mislevy, R.J. (2018). Sociocognitive foundations of educational measurement. Routledge. https://doi.org/10.4324/9781315269473
24. Molnar, C. (2025). Interpretable machine learning: A guide for making black box models explainable (3rd ed.). https://christophm.github.io/interpretable-ml-book
25. Nadeau, C., & Bengio, Y. (2003). Inference for the generalization error. Machine Learning, 52(3), 239–281. https://doi.org/10.1023/A:1024068626366
26. OECD. (2009). PISA data analysis manual: SPSS (2nd ed.). OECD Publishing. https://doi.org/10.1787/9789264056275-en
27. OECD. (2023). PISA 2022 assessment and analytical framework. OECD Publishing. https://doi.org/10.1787/dfe0bf9c-en
28. Öz, E., Bulut, O., Cellat, Z.F., & Yürekli, H. (2025). Stacking: An ensemble learning approach to predict student performance in PISA 2022. Education and Information Technologies, 30(6), 7753–7779. https://doi.org/10.1007/s10639-024-13110-2
29. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830. https://jmlr.org/papers/v12/pedregosa11a.html
30. Peña-Ayala, A. (2014). Educational data mining: A survey and a data mining-based analysis of recent works. Expert Systems with Applications, 41(4), 1432 1462. https://doi.org/10.1016/j.eswa.2013.08.042
31. Probst, P., Wright, M.N., & Boulesteix, A.L. (2019). Hyperparameters and tuning strategies for random forest. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(3), e1301. https://doi.org/10.1002/widm.1301
32. Romero, C., & Ventura, S. (2020). Educational data mining and learning analytics: An updated survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(3), Article e1355. https://doi.org/10.1002/widm.1355
33. Rutkowski, L., Gonzalez, E., Joncas, M., & von Davier, M. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher, 39(2), 142–151. https://doi.org/10.3102/0013189X10363170
34. Saito, T., & Rehmsmeier, M. (2015). The precision–recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3), Article e0118432. https://doi.org/10.1371/journal.pone.0118432
35. Seni, G., & Elder, J.F. (2010). Ensemble methods in data mining: Improving accuracy through combining predictions. Morgan & Claypool Publishers. https://doi.org/10.2200/S00240ED1V01Y200912DMK002
36. Sill, J., Takács, G., Mackey, L., & Lin, D. (2009). Feature-weighted linear stacking [Preprint]. arXiv. https://arxiv.org/abs/0911.0460
37. Slater, S., Joksimovic, S., Kovanović, V., Baker, R.S., & Gasevic, D. (2017). Tools for educational data mining: A review. Journal of Educational and Behavioral Statistics, 42(1), 85–106. https://doi.org/10.3102/1076998616666808
38. Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427 437. https://doi.org/10.1016/j.ipm.2009.03.002
39. Ting, K.M., & Witten, I.H. (1999). Issues in stacked generalization. Journal of Artificial Intelligence Research, 10, 271–289. https://doi.org/10.1613/jair.594
40. van der Laan, M.J., Polley, E.C., & Hubbard, A.E. (2007). Super learner. Statistical Applications in Genetics and Molecular Biology, 6(1), Article 25. https://doi.org/10.2202/1544-6115.1309
41. Vanschoren, J. (2019). Meta-learning. In F. Hutter, L. Kotthoff, & J. Vanschoren (Eds.), Automated machine learning: Methods, systems, challenges (pp. 35–61). Springer. https://doi.org/10.1007/978-3-030-05318-5_2
42. Wang, X., Perry, L.B., & Malpique, A. (2023). Factors predicting mathematics achievement in PISA: A systematic review. Large scale Assessments in Education, 11, Article 24. https://doi.org/10.1186/s40536-023-00174-8
43. Witten, I.H., Frank, E., Hall, M.A., & Pal, C.J. (2016). Data mining: Practical machine learning tools and techniques (4th ed.). Morgan Kaufmann. https://doi.org/10.1016/C2015-0-02071-8
44. Wolpert, D.H. (1992). Stacked generalization. Neural Networks, 5(2), 241 259. https://doi.org/10.1016/S0893-6080(05)80023-1
45. Zhang, S. (2012). Nearest neighbor selection for iteratively kNN imputation. Journal of Systems and Software, 85(11), 2541–2552. https://doi.org/10.1016/j.jss.2012.05.073