The Effects of Log Data on Students’ Performance

Abstract

This study aimed to assess the relationships between response times (RTs), the number of actions taken to solve a given item, and student performance. In addition, the interaction between the students’ information and communications technology (ICT) competency, reading literacy, and log data (time and number of actions) were examined in order to gain additional insights regarding the relations between student performance and log data. The sample consisted of 2 348 students who participated in the triennial international large-scale assessment of the Programme for International Student Assessment (PISA). For the current study, 18 items in the one cluster of the 91st booklet were chosen. To achieve the aim of the study, explanatory item response modeling (EIRM) framework based on generalized linear mixed modeling (GLMM) was used. The results of this study showed that students who spent more time on items and those that took more actions on items were more likely to answer the items correctly. However, this effect did not have variability across items and students. Moreover, the interaction only with reading and the number of actions was found to have a positive effect on the students’ overall performance.

Keywords

• Test-taking behaviors,explanatory item response modelling,log data,technology-based assessment

References

1. Azzolini, D., Bazoli, N., Lievore, I., Schizzerotto, A., & Vergolini, L. (2019). Beyond achievement. A comparative look into 15-year-olds’ school engagement, effort and perseverance in the European Union. Luxembourg: Publications Office of the European Union.
2. Bates, D., & Maechler, M. (2014). lme4: Linear mixed-effects models using S4 classes. R package. Available from http://CRAN.R-project.org/package=lme4.
3. Briggs, D. C. (2008). Using explanatory item response models to analyze group differences in science achievement. Applied Measurement in Education, 21(2), 89-118. doi: 10.1080/08957340801926086
4. De Ayala, R. J. (2009). The theory and practice of Item Response Theory. New York: The Guilford Press.
5. De Boeck, P., & Wilson, M. (Eds.). (2004). Explanatory item response models: A generalized linear and nonlinear approach. New York, NY: Springer, doi: 10.1007/978-1-4757-3990-9
6. De Boeck, P., Bakker, M., Zwitser, R., Nivard, M., Hofman, A., Tuer- linckx, F., & Partchev, I. (2011). The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software, 39, 1-28.
7. Dodonova Y. A., & Dodonov Y. S. (2013). Faster on easy items, more accurate on difficult ones: Cognitive ability and performance on a task of varying difficulty. Intelligence, 41, 1–10. doi: 10.1016/j.intell.2012.10.003
8. Goldhammer, F., Naumann, J., & Greiff, S. (2015). More is not always better: The relation between item response and item response time in Raven’s matrices. Journal of Intelligence, 3(1), 21-40.

9. Goldhammer, F., Naumann, J., Stelter, A., Tóth, K., Rölke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106(3), 608.
10. Goldhammer, F., & Klein Entink, R. (2011). Speed of reasoning and its relation to reasoning ability. Intelligence, 39, 108-119.
11. Greiff, S., Niepel, C., Scherer, R., & Martin, R. (2016). Understanding students' performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files. Computers in Human Behavior, 61, 36-46.
12. Greiff, S., Wüstenberg, S., & Avvisati, F. (2015). Computer-generated log-file analyses as a window into students' minds? A showcase study based on the PISA 2012 assessment of problem solving. Computers & Education, 91, 92-105.
13. Greiff, S., Wüstenberg, S., Csapo, B., Demetriou, A., Hautamaki, J., Graesser, A. C., et al. (2014). Domain-general problem solving skills and education in the 21st century. Educational Research Review, 13,74-83.
14. Klein Entink, R. H., Fox, J. P., & van der Linden, W. J. (2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74, 21-48. doi: 10.1007/s11336-008-9075-y
15. Kong, X., Wise, S. L., & Bhola, D. S. (2007). Setting the response time threshold parameter to differentiate solution behavior from rapid-guessing behavior. Educational and Psychological Measurement, 67, 606–619.
16. Kyllonen, P., & Zu, J. (2016). Use of response time for measuring cognitive ability. Journal of Intelligence, 4(4), 14.
17. Lasry, N., Watkins, J., Mazur, E., & Ibrahim, A. (2013). Response times to conceptual questions. American Journal of Physics, 81, 703. doi:10.1119/1.4812583.
18. Lee, Y. H. & Haberman, S. J. (2016) Investigating test-taking behaviors using timing and process data. International Journal of Testing, 16(3), 240-267, doi: 10.1080/15305058.2015.1085385.
19. Lee, Y. H., & Jia, Y. (2014). Using response time to investigate students' test-taking behaviors in a NAEP computer-based study. Large-scale Assessments in Education, 2(1), 8.
20. Meyer, P. J. (2010). A mixture Rasch model with response time components. Applied Psychological Measurement, 34, 521–538.
21. OECD (2017a). PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic, Financial Literacy and Collaborative Problem Solving. PISA,
22. OECD Publishing, Paris, https://doi.org/10.1787/9789264281820-en.
23. OECD (2017b). PISA 2015 results (Volume V): Collaborative problem solving. PISA, OECD Publishing, Paris.
24. OECD (2017c). PISA 2015 technical report. PISA, OECD Publishing, Paris.
25. Petscher, Y., Mitchell, A. M., & Foorman, B. R. (2015). Improving the reliability of student scores from speeded assessments: An illustration of conditional item response theory using a computer-administered measure of vocabulary. Reading and Writing, 28,31–56. doi:10.1007/s11145-014-9518-z
26. R Core Team (2018). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.
27. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danish Institute for Educational Research
28. Rijmen, F., Tuerlinckx, F., De Boeck, P., & Kuppens, P. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8(2), 185–205.
29. Rios, J. A., Guo, H., Mao, L., & Liu, O. L. (2017). Evaluating the impact of careless responding on aggregated-scores: To filter unmotivated examinees or not? International Journal of Testing, 17,74–104. doi:10.1080/15305058.2016.1231193
30. Robitzsch, A. (2019). Package “sirt”. (https://cran.r-project.org/web/packages/sirt/sirt.pdf)
31. Schnipke, D. L., & Scrams, D. J. (2002). Exploring issues of examinee behavior: Insights gained from response-time analyses. Computer-based testing: Building the foundation for future assessments, 237-266. in Computer-Based Testing: Building the Foundation for Future Assessments, edited by C. N. Mills et al. (Lawrence Erlbaum Associates, Mahwah, NJ, 1997).
32. Su, S., & Davison, M. L. (2019) Improving the Predictive Validity of Reading Comprehension Using Response Times of Correct Item Responses. Applied Measurement in Education, 32 (2), 166-182, doi: 10.1080/08957347.2019.1577247
33. van der Linden, W. J. (2009). Conceptual issues in response‐time modeling. Journal of Educational Measurement, 46(3), 247-272.
34. Verbić, S., & Tomić, B. (2009). Test item response time and the response likelihood. Retrieved on 20 March 2019 from http://arxiv.org/ftp/arxiv/papers/0901/0901.4356.pdf .
35. Wickham, H. (2012). reshape2: Flexibly reshape data: a reboot of the reshape package. R package version, 1(2).
36. Wickham, H., Chang, W., & Wickham, M. H. (2016). Package ‘ggplot2’. Create Elegant Data Visualisations Using the Grammar of Graphics. Version, 2(1), 1-189.Wilson, M., De Boeck, P., & Carstensen, C. H. (2008). Explanatory Item Response Models: A brief introduction. In J Hartig, E Klieme, & D Leutner (Eds.), Assessment of competencies in educational contexts (pp. 91–120). Cambridge, MA: Hogrefe.
37. Wise, S. L. (2006). An investigation of the differential effort received by items on a low-stakes, computer-based test. Applied Measurement in Education, 19, 95-114.
38. Wise, S. L., & DeMars, C. E. (2005). Examinee motivation in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10, 1–18.
39. Wise, S. L., Kingsbury, G. G., Thomason, J., & Kong, X. (2004). An investigation of motivation filtering in a statewide achievement testing program. Paper presented at the Annual meeting of the Na- tional Council on Measurement in Education, San Diego, CA.
40. Wang, C., & Xu, G. (2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68(3), 456–477. doi:10.1111/bmsp.12054
41. Wu, A. D., Chen, M. Y., & Stone, J. E. (2018). Investigating How Test-Takers Change Their Strategies to Handle Difficulty in Taking a Reading Comprehension Test: Implications for Score Validation. International Journal of Testing, 18(3), 253-275.
42. Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145.