Investigation of item parameter drift of anchor items in TIMSS practices

Abstract

This study examines item parameter drift (IPD) in eighth-grade science anchor items from the paper-based cycles of the Trends in International Mathematics and Science Study (TIMSS) in 2011, 2015, and 2019. Korea, Italy, and Saudi Arabia were selected as examples of high-, medium-, and low-achieving education systems to explore how IPD develops across different performance contexts. Using the three-parameter logistic model (3PLM) estimated in Lexter and D² statistics computed in R, the study evaluated changes in item parameters across years and country pairs for 30 multiple-choice anchor items. The results showed that eight items exhibited temporal IPD in at least one comparison period, with most drift concentrated in the 2015–2019 and 2011–2019 contrasts. Drift occurred mainly in chemistry items at the applying cognitive level, alongside a smaller number of biology and physics items. Cross-country analyses identified 18 items with IPD in at least one country pair, with the largest number of drifting items observed between Korea and Saudi Arabia. Items involving symbolic chemical notation and representational demands were especially likely to drift across systems that differ in achievement level and opportunity to learn. These findings indicate that IPD in TIMSS science is concentrated in a subset of anchor items and is shaped jointly by temporal change and cross-national differences. The study underscores the importance of routine IPD monitoring, targeted review of repeatedly drifting items, and careful anchor selection to support valid cross-cycle and cross-national comparisons in large-scale assessments.

Keywords

• Item parameter drift
• TIMSS
• Science assessments
• Item response theory
• Cross-national comparisons

References

1. Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716-723. https://doi.org/10.1109/TAC.1974.1100705
2. Babcock, B., & Albano, A.D. (2012). Rasch scale stability in the presence of item parameter and trait drift. Applied Psychological Measurement, 36(7), 565 580. https://doi.org/10.1177/0146621612455090
3. Bock, D.B, Muraki, E., & Pfeiffenberger, W. (1988). Item pool maintenance in the presence of item parameter drift. Journal of Educational Measurement, 25(4), 275 285. https://doi.org/10.1111/j.1745-3984.1988.tb00308.x
4. De Ayala, R.J. (2022). The theory and practice of item response theory (2nd ed.). Guilford Press.
5. Deng, H., & Melican, G. (2010). An investigation of scale drift in computer adaptive test. (Report no. 2010-2). The College Board. https://files.eric.ed.gov/fulltext/ED561045.pdf
6. Fishbein, B., Martin, M.O., Mullis, I.V.S, & Foy , P. (2018). The TIMSS 2019 item equivalence study: Examining mode effects for computer-based assessment and implications for measuring trends. Large-scale Assessments in Education, 6(11), 1-23. https://doi.org/10.1186/s40536-018-0064-z
7. Glas, C., & Van Buuren, N. (2021). Lexter Manual. https://shinylexter.com/
8. Goldstein, H. (1983). Measuring changes in educational attainment over time: Problems and possibilities. Journal of Educational Measurement, 20(4), 369 377. https://www.jstor.org/stable/1434953

9. Gonulates, E. (2024). Package ‘irt’ [Computer software manual]. https://github.com/egonulates/irt.
10. He, Y., & Cui, Z. (2020). Evaluating robust scale transformation methods with multiple outlying common items under IRT true score equating. Applied Psychological Measurement, 44(4), 296-310. https://doi.org/10.1177/0146621619886050
11. Huynh, H., & Meyer, P. (2010). Use of robust z in detecting unstable items in item response theory models. Practical Assessment, Research, and Evaluation, 15(2), 1 7. https://doi.org/10.7275/ycx6-e864
12. International Association for the Evaluation of Educational Achievement (IEA) (2019). TIMSS 2019 international database. https://timss2019.org/international-database/
13. International Association for the Evaluation of Educational Achievement (IEA) (2023). TIMSS 2023 international results in mathematics and science. https://timss2023.org/results/
14. Jerrim, J., Micklewright, J., Heine, J.H., Salzer, C., & McKeown, C. (2018). PISA 2015: How big is the ‘mode effect’ and what has been done about it? Oxford Review Education, 44(4), 476 493. https://doi.org/10.1080/03054985.2018.1430025
15. Jurich, D., & Liu, C. (2023) Detecting item parameter drift in small sample rasch equating. Applied Measurement in Education, 36(4), 326-339. https://doi.org/10.1080/08957347.2023.2274567
16. Kolen, M.J., & Brennan, R.L. (2014). Test equating, scaling, and linking: Methods and practices (3rd ed.). Springer Science + Business Media. https://doi.org/10.1007/978-1-4939-0317-7
17. Kuha, J. (2004). AIC and BIC: Comparisons of assumptions of performance. Sociological Methods and Research, 33(2), 188-229. https://doi.org/10.1177/0049124103262065
18. Lee, H.S., & Geisinger K.F. (2019) Item parameter drift in context questionnaires from international large scale assessments. International Journal of Testing, 19(1), 23 51. https://doi.org/10.1080/15305058.2018.1481852
19. Li, X. (2008). An investigation of the item parameter drift in the examination for the certificate of proficiency in English (ECPE). Spaan Fellow Working Papers in Second or Foreign Language Assessment, 6, 1-28.
20. Li, Y. (2012). Examining the impact of drifted polytomous anchor items on test characteristic curve (TCC) Linking and IRT true score equating. ETS Research Report Series, 2012(1), i 22. https://doi.org/10.1002/j.2333-8504.2012.tb02291.x
21. Liou, P.Y., & Bulut, O. (2017). The effects of item format and cognitive domain on students’ science performance in TIMSS 2011. Research in Science Education, 50, 99 21. https://doi.org/10.1007/s11165-017-9682-7
22. Martin, M.O., & Kelly, D.L. (1998). Third international mathematics and science study, technical report (Volume 3): Implementation and analysis. Boston College.
23. Martin, M.O., & Mullis, I.V.S. (Eds.). (2012). Methods and procedures in TIMSS and PIRLS 2011. TIMSS & PIRLS International Study Center, Boston College.
24. Mullis, I.V.S, Martin, M.O., Foy, P., Kelly, D., & Fishbein, B. (2020). TIMSS 2019 international results in mathematics and science. TIMSS & PIRLS International Study Center.
25. Park, Y.S, Lee, Y.S., & Xing, K. (2016). Investigating the impact of item parameter drift for item response theory models with mixture distributions. Frontiers in Psychology, 7(255). https://doi.org/10.3389/fpsyg.2016.00255
26. Schwarz, G.E. (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461 464. https://www.jstor.org/stable/2958889
27. Solano-Flores, G., & Li, M. (2009). Generalizability of cognitive interview-based measures across cultural groups. Educational Measurement: Issues and Practice, 28(1), 9 18. https://doi.org/10.1111/j.1745-3992.2009.00143.x
28. Uysal, İ., Şahin Kürşad, M., & Kılıç, A.F. (2022). Effect of item parameter drift in mixed format common items on test equating. Participatory Educational Research, 9(5), 143 160. https://doi.org/10.17275/per.22.108.9.5
29. Wells, C.S., & Bolt, D.M. (2008). Investigation of a nonparametric procedure for assessing goodness-of fit in item response theory. Applied Measurement in Education, 21(1), 22 40. https://doi.org/10.1080/08957340701796464
30. Wells, C.S., Hambleton, R.K., Kirkpatrick, R., & Meng, Y. (2014). An examination of two procedures for identifying consequential item parameter drift. Applied Measurement in Education, 27(3), 214-231. https://doi.org/10.1080/08957347.2014.D905786
31. Wu, A.D., Li, Z., Ng., S.L., & Zumbo, B.D. (2006, May 21-26). Investigating and comparing the item parameter drift in the mathematics anchor/trend items in TIMSS between Singapore and the United States [Paper presentation]. The 32nd Annual Conference in International Association for Educational Assessment, Singapore, Singapore.