Items, gender, response time, and styles in play: Exploring student-item interactions in mathematics literacy

Öz

This study examines how student characteristics, item properties, and their interactions influence mathematics literacy using data from 7,250 Turkish students in the Programme for International Student Assessment (PISA) 2022. Employing Explanatory Item Response Modeling (EIRM), the analysis incorporated individual-level factors such as response styles (acquiescent, midpoint, extreme, and disacquiescent) and gender, along with item-level variables including format, cognitive demand, and response time behavior (solution-oriented vs. rapid guessing). Results indicate that acquiescent and midpoint response styles are positively associated with performance, while extreme response style is negatively linked to performance. Male students outperformed females. Open-ended and reasoning-level questions were more challenging than multiple-choice or other cognitive-demand items. Significant interactions emerged between response styles and time-related behaviors. Beyond these empirical findings, this study contributes conceptually by positioning response styles, traditionally associated with attitudinal measures, within cognitive performance. The results carry implications for the design of large-scale assessments, highlighting the importance of accounting for both cognitive and behavioral factors in interpreting test performance.

Anahtar Kelimeler

• Mathematics literacy
• Response styles
• Response time
• EIRM

References

1. Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716-723. https://doi.org/10.1109/TAC.1974.1100705
2. Benítez, I., He, J., Van de Vijver, F.J.R., & Padilla, J.L. (2016). Linking extreme response style to response processes: A cross-cultural mixed methods approach. International Journal of Psychology, 51(6), 464-473. https://doi.org/10.1002/ijop.12379
3. Bulut, O., Cormier, D.C., & Shin, J. (2020). An intelligent recommender system for personalized test administration scheduling with computerized formative assessments. Frontiers in Education, 5, Article 572612. 1-11. http://dx.doi.org/10.3389/feduc.2020.572612
4. Bulut, O., Gorgun, G., & Yildirim-Erbasli, S.N. (2021). Estimating explanatory extensions of dichotomous and polytomous Rasch Models: The eirm package in R. Psych, 3(3), 308 321. https://doi.org/10.3390/psych3030023
5. Chalmers R.P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48, 1-29. https://doi.org/10.18637/jss.v048.i06
6. Cohen, J. (2013). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge.
7. Deribo, T., Goldhammer, F., & Kroehne, U. (2023). Changes in the speed-ability relation through different treatments of rapid guessing. Educational and Psychological Measurement, 83(3), 473-494. https://doi.org/10.1177/00131644221109490
8. Ding, H., & Homer, M. (2020). Interpreting mathematics performance in PISA: Taking account of reading performance. International Journal of Educational Research, 102, Article 101566. https://doi.org/10.1016/j.ijer.2020.101566

9. Eccles, J.S. (2011). Gendered educational and occupational choices: Applying the Eccles et al. model of achievement-related choices. International Journal of Behavioral Development, 35(3), 195-201. https://doi.org/10.1177/0165025411398185
10. Falk, C.F. & Ju, U. (2020). Estimation of response styles using the multidimensional nominal response model: A tutorial and comparison with sum scores. Frontiers in Psychology, 11, Article 72. https://doi.org/10.3389/fpsyg.2020.00072
11. Gómez-Chacón, I.M., García-Madruga, J.A., Vila, J. Ó., Elosúa, M.R., & Rodríguez, R. (2014). The dual processes hypothesis in mathematics performance: Beliefs, cognitive reflection, working memory and reasoning. Learning and Individual Differences, 29, 67 73. https://doi.org/10.1016/j.lindif.2013.10.001
12. Henninger, M., & Plieninger, H. (2021). Different styles, different times: How response times can inform our knowledge about the response process in rating scale measurement. Assessment, 28(5), 1301-1319. https://doi.org/10.1177/1073191119900003
13. He, J., & Van de Vijver, F.J. (2013). A general response style factor: Evidence from a multi-ethnic study in the Netherlands. Personality and Individual Differences, 55(7), 794-800.
14. Hibbing, M.V., Cawvey, M., Deol, R., Bloeser, A.J., & Mondak, J.J. (2019). The relationship between personality and response patterns on public opinion surveys: The big five, extreme response style, and acquiescence response style. International Journal of Public Opinion Research, 31(1), 161-177. https://doi.org/10.1093/ijpor/edx005
15. İlhan, M., Boztunç Öztürk, N., & Şahin, M.G. (2020). The effect of the item’s type and cognitive level on its difficulty index: The sample of TIMSS 2015. Participatory Educational Research, 7(2), 47-59. https://doi.org/10.17275/per.20.19.7.2
16. Li, F., Cohen, A.S., Kim, S.H., & Cho, S.J. (2009). Model selection methods for mixture dichotomous IRT models. Applied Psychological Measurement, 33(5), 353 373. https://doi.org/10.1177/0146621608326422
17. Liu, O.L., & Wilson, M. (2009). Gender differences in large-scale math assessments: PISA trend 2000 and 2003. Applied Measurement in Education, 22(2), 164 184. https://doi.org/10.1080/08957340902754635
18. Lüdecke, D. (2018). ggeffects: Tidy data frames of marginal effects from regression models. Journal of Open Source Software, 3(26), Article 772. https://doi.org/10.21105/joss.00772
19. Meisenberg, G., & Williams, A. (2008). Are acquiescent and extreme response styles related to low intelligence and education? Personality and Individual Differences, 44(7), 1539 1550. https://doi.org/10.1016/j.paid.2008.01.010
20. Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176. https://doi.org/10.1177/014662169201600206
21. Oberleiter, S., Fries, J., Schock, L.S, Steininger, B., & Pietschnig, J. (2023). Predicting cross-national sex differences in large-scale assessments of students’ reading literacy, mathematics, and science achievement: Evidence from PIRLS and TIMSS. Intelligence, 100. Article 101784. https://doi.org/10.1016/j.intell.2023.101784
22. OECD. (2019). PISA 2018 results (Volume I): What students know and can do. OECD Publishing. https://doi.org/10.1787/5f07c754-en
23. OECD (2023). PISA 2022 technical report. OECD Publishing.
24. Özer Özkan, Y., & Özaslan, N. (2018). Student achievement in Turkey, according to question types used in PISA 2003-2012 mathematic literacy tests. International Journal of Evaluation and Research in Education (IJERE), 7(1), 57. https://doi.org/10.11591/ijere.v1i1.11045
25. Paas, F., & Ayres, P. (2014). Cognitive load theory: A broader view on the role of memory in learning and education. Educational Psychology Review, 26(2), 191-195. https://doi.org/10.1007/s10648-014-9263-5
26. Palmer, C. (2011). Interpretation of β in log linear models. https://web.mit.edu/cjpalmer/www/beta_in_log-linear_regression.pdf
27. R Core Team. (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing.
28. Reilly, D., Neumann, D.L., & Andrews, G. (2015). Sex differences in mathematics and science achievement: A meta-analysis of national assessment of educational progress assessments. Journal of Educational Psychology, 107(3), 645-662. https://doi.org/10.1037/edu0000012
29. Retnawati, H., & Wulandari, N.F. (2019). The development of students’ mathematical literacy proficiency. Problems of Education in the 21st Century, 77(4), 502 514. https://doi.org/10.33225/pec/19.77.502
30. Rios, J.A., Guo, H., Mao, L., & Liu, O.L. (2017). Evaluating the impact of careless responding on aggregated-scores: To filter unmotivated examinees or not? International Journal of Testing, 17(1), 74-104. https://doi.org/10.1080/15305058.2016.1231193
31. Saris, W.E., Revilla, M., Krosnick, J.A., & Shaeffer, E.M. (2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods, 4(1), 61-79. https://doi.org/10.18148/srm/2010.v4i1.2682
32. Schnipke, D.L., & Scrams, D.J. (1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34(3), 213-232. https://doi.org/10.1111/j.1745-3984.1997.tb00516.x
33. Soland, J. (2018a). Are achievement gap estimates biased by differential student test effort? Putting an important policy metric to the test. Teachers College Record, 120(12), 1 26. https://doi.org/10.1177/0161468118120012
34. Soland, J. (2018b). The achievement gap or the engagement gap? Investigating the sensitivity of gaps estimates to test motivation. Applied Measurement in Education, 31(4), 312 323. https://doi.org/10.1080/08957347.2018.1495213
35. Stacey, K., & Turner, R. (2015). The evolution and key concepts of the PISA mathematics frameworks. In K. Stacey & R. Turner (Eds.), Assessing Mathematical Literacy: The PISA Experience (pp. 5-33). Springer.
36. Stoet, G., & Geary, D.C. (2013). Sex differences in mathematics and reading achievement are inversely related: Within and across nation assessment of 10 years of PISA data. PLoS ONE, 8(3). https://doi.org/10.1371/journal.pone.0057988
37. Schwarz, G.E. (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461 464. https://doi.org/10.1214/aos/1176344136
38. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257-285. https://doi.org/10.1207/s15516709cog1202_4
39. Tarim, K., & Tarku, H. (2022). Investigation of the questions in 8th grade mathematics textbook in terms of mathematical literacy. International Electronic Journal of Mathematics Education, 17(2), Article em0682. https://doi.org/10.29333/iejme/11819
40. Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/CBO9780511819322
41. Ulitzsch, E., Pohl, S., Khorramdel, L., Kroehne, U., & von Davier, M. (2023). Using response times for joint modeling of careless responding and attentive response styles. Journal of Educational and Behavioral Statistics, 49(2).173-206. https://doi.org/10.3102/10769986231173607
42. Wei, W., Yuan, H., Chen, C., & Zhou, X. (2011). Cognitive correlates of performance in advanced mathematics. British Journal of Educational Psychology, 82(1), 157 181. https://doi.org/10.1111/j.2044-8279.2011.02049.x
43. Weijters, B., Geuens, M., & Schillewaert, N. (2010). The individual consistency of acquiescence and extreme response style in self-report questionnaires. Applied Psychological Measurement, 34(2), 105-121. https://doi.org/10.1177/0146621609338593
44. Weijters, B., Schillewaert, N., & Geuens, M. (2008). Assessing response styles across modes of data collection. Journal of the Academy of Marketing Science, 36, 409 422. https://doi.org/10.1007/s11747-007-0077-6
45. Wetzel, E., Carstensen, C.H., & Böhnke, J.R. (2013). Consistency of extreme response style and non-extreme response style across traits. Journal of Research in Personality, 47(2), 178 189. https://doi.org/10.1016/j.jrp.2012.10.010
46. Wetzel, E., Lüdtke, O., Zettler, I., & Böhnke, J.R. (2016). The stability of extreme response style and acquiescence over 8 years. Assessment, 23(3), 279-291. https://doi.org/10.1177/1073191115583714
47. Wilson, M., & De Boeck, P. (2004). Descriptive and explanatory item response models. In P. De Boeck & M. Wilson (Eds.), Explanatory item response models: A generalized linear and nonlinear approach. Springer-Verlag.
48. Wise, S.L. (2017). Rapid-guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice, 36(4), 52-61. https://doi.org/10.1111/emip.12165
49. Wise, S.L., & Ma, L. (2012, April 14-16). Setting response time thresholds for a CAT item pool: The normative threshold method [Paper presentation]. The Annual Meeting of the National Council on Measurement in Education, Vancouver, Canada.
50. Wise, S.L., & Kingsbury, G.G. (2016). Modeling student test-taking motivation in the context of an adaptive achievement test. Journal of Educational Measurement, 53(1), 86 105. https://doi.org/10.1111/jedm.12102