Detection of differential item functioning with latent class analysis: PISA 2018 mathematical literacy test

Abstract

The purpose of this research is to determine whether PISA 2018 mathematical literacy test items show a differential item functioning across countries. For this purpose, only the items in booklet number three were examined using the MIMIC method with Latent Class Analysis (LCA) approach. PISA 2018 tests are mostly developed in English. Therefore, in DIF analyses, the reference group is the UK, while the focal groups consist of the other countries examined in the research (Türkiye, Finland, Japan, and the USA). According to the results, of the 23 test items, statistically significant DIF was observed in eight items in the UK-Türkiye sample, in seven items in the UK-Finland sample, in eleven items in the UK-Japan sample, and in three items in the UK-USA sample. It is seen that the effect and size of DIF in non-homogeneous groups differ between groups and these effects can be examined in more detail with the LCA method.

Keywords

• Cross country DIF
• Finite mixture models
• MIMIC
• Latent class analysis
• PISA

References

1. Camilli, G., & Shepard, L.A. (1994). Methods for identifying biased test items. SAGE Publications.
2. Clauser, B.E., & Mazor, K.M. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues and Practice, 17(1), 31 - 44.
3. Elkonca, F. (2020). ABİDE öz yeterlilik ölçeği DMF kaynaklarının gizil sınıf yaklaşımıyla incelenmesi [An analysis of the DIF sources of ABİDE self-efficacy scale by means of a latent class approach] [Unpublished doctoral dissertation]. Gazi University.
4. Fraenkel, J.R., Wallen, N.E., & Hyun, H.H. (2011). How to design and evaluate research in education. McGraw-Hill Education.
5. Güngör Culha, D. (2012). Örtük sınıf analizlerinde ölçme eşdeğerliğinin incelenmesi [Investigating measurement equivalence with latent class analysis] [Unpublished doctoral dissertation]. Ege University.
6. Hambleton, R.K., Merenda, P.F., & Spielberger, C.D. (Eds.). (2005). Adapting educational and psychological tests for cross-cultural assessment. Lawrence Erlbaum Associates.
7. Harrington, D. (2009). Confirmatory factor analysis. Oxford University Press.
8. Kerlinger, F.N. (1999). Foundations of behavioral research. Wadsworth Publishing.

9. Lanza, S., & Collins, L. (2009). Latent class and latent transition analysis: With applications in the social, behavioral, and health sciences. John Wiley & Sons, Inc.
10. Magidson, J., & Vermunt, J.K. (2004). Latent class models. D. Kaplan (Eds.), The sage handbook of quantitative methodology for the social sciences (s. 175-198). Sage Publications.
11. Masyn, K.E. (2017). Measurement invariance and differential item functioning in latent class analysis with stepwise multiple indicator multiple cause modeling. Structural Equation Modeling: A Multidisciplinary Journal, 24(2) 180-197.
12. McCutcheon, A.L. (1987). Latent class analysis. Sage Publication.
13. MEB (2019). PISA 2018 Türkiye ön raporu [PISA 2018 Results]. T.C. Milli Eğitim Bakanlığı.
14. Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Research Article, 18(2), 5-11.
15. Nunnally, J.C., & Bernstein, I.H. (1994). Psychometric theory third edition. McGraw-Hill.
16. Nylund-Gibson, K., Grimm, R., Quirk, M., & Furlong, M. (2014). A latent transition mixture model using the three-step specification. Structural Equation Modeling: A Multidisciplinary Journal, 21(3), 439-454.
17. OECD. (2016a). Sampling in PISA. OECD Publishing.
18. OECD. (2016b). PISA 2018 technical report. OECD Publishing.
19. OECD. (2016c). PISA 2018 translation and adaptation guidelines. OECD Publishing.
20. OECD. (2019). PISA 2018 mathematics framework. PISA 2018 assessment and analytical framework (s. 73-95). OECD Publishing. https://doi.org/10.1787/13c8a22c-en
21. Oliveri, M.E., Ercikan, K., Lyons-Thomas, J., & Holtzman, S. (2016). Analyzing fairness among linguistic minority populations using a latent class differential item functioning approach. Applied Measurement in Education, 29(1), 17 29. https://doi.org/10.1080/08957347.2015.1102913
22. Saatçioğlu, F.M. (2022). Differential item functioning across gender with MIMIC modeling: PISA 2018 financial literacy items. International Journal of Assessment Tools in Education, 9(3), 631-653. https://doi.org/10.21449/ijate.1076464
23. Sawatzky, R., Russell, L.B., Sajobi, T.T., Lix, L.M., Kopec, J., & Zumbo, B.D. (2018). The use of latent mixture models to identify Invariant Items in test construction. Qual Life Res, 27(7), 1745-1755. https://doi.org/10.1007/s11136-017-1680-8
24. Tsaousis, I., Sideridis, G.D., & AlGhamdi, H.M. (2020). Measurement invariance and differential item functioning across gender within a latent class analysis framework: Evidence from a high-stakes test for university admission in Saudi Arabia. Frontiers in Psychology, 11(622). https://doi.org/10.3389/fpsyg.2020.00622
25. Uyar, Ş. (2020). Latent class approach to detect differential item functioning: PISA 2015. Eurasian Journal of Educational Research, 20(88), 179-198. https://doi.org/10.14689
26. Vermunt, J.K. (2010). Latent class modeling with covariates: Two improved three-step approaches. Political Analysis, 18(4), 450-469.
27. Zumbo, B.D. (1999). A handbook on the theory and methods of differential item functioning (DIF): logistic regression modeling as a unitary framework for binary and Likert-Type (Ordinal) item scores. ON: Directorate of Human Resources Research and Evaluation.
28. Zumbo, B.D., Liu, Y., Wu, A.D., Shear, B.R., Olvera Astivia, O.L., & Ark, T.K. (2015). A methodology for Zumbo’s third generation DIF analyses and the ecology of item responding. Language Assessment Quarterly, 12(1), 136 151. https://doi.org/10.1080/15434303.2014.972559