Examination of response time effort in TIMSS 2019: Comparison of Singapore and Türkiye

Abstract

In this paper, it is aimed to evaluate different aspects of students' response time to items in the mathematics test and their test effort as an indicator of test motivation with the help of some variables at the item and student levels. The data consists of 4th-grade Singapore and Turkish students participating in the TIMSS 2019. Response time was examined in terms of item difficulties, content and cognitive domains of the items in the mathematics test self-efficacy for computer use, home resources for learning, confident in mathematics, like learning mathematics, and gender variables at the student level. In the study, it was determined that all variables considered at the item level affected the response time of the students in both countries. It was concluded that the amount of variance explained by the student-level variables in the response time varied for each the country. Another finding of the study showed that the cognitive level of the items positively related to the mean response time. Both Turkish and Singaporean students took longer to respond to data domain items compared to number and measurement and geometry domain items. Additionally, based on the criterion that the response time effort index was less than .8, rapid-guessing behavior, and therefore low motivation, was observed below 1% for both samples. Besides, we observed that Turkish and Singaporean students were likely to have rapid guessing behavior when an item in the reasoning domain became increasingly difficult. A similar result was identified in the data content domain, especially for Turkish graders.

Keywords

• Response time effort
• Rapid guessing behavior
• Solution behavior
• Cognitive domain
• Content domain

References

1. American Psychological Association. (2022). Self-report bias. In APA dictionary of psychology. https://dictionary.apa.org/self-report-bias
2. Barry, C.L, & Finney, S.J. (2009). Exploring change in test-taking motivation. Northeastern Educational Research Association
3. Barry, C.L., Horst, S.J., Finney, S.J., Brown, A.R., & Kopp, J.P. (2010). Do examinees have similar test-taking effort? A high-stakes question for low-stakes testing. International Journal of Testing, 10, 342–363. https://doi.org/10.1080/15305058.2010.508569
4. Baumert, J., & Demmrich, A. (2001). Test motivation in the assessment of student skills: the effects of incentives on motivation and performance. European Journal of Psychology of Education, 14, 441–462. http://www.jstor.org/stable/23420343
5. Bennett, R.E., Brasell, J., Oranje, A., Sandene, B., Kaplan, K., & Yan, F. (2008). Does it matter if I take my mathematics test on a computer? A second empirical study of mode effects in NAEP. Journal of Technology, Learning, and Assessment, 6(9), 1 39. https://files.eric.ed.gov/fulltext/EJ838621.pdf
6. Bergstrom, B.A., Gershon, R.C., & Lunz, M.E. (1994, April 4-8). Computer adaptive testing: Exploring examinee response time using hierarchical linear modeling. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA. https://files.eric.ed.gov/fulltext/ED400287.pdf
7. Borgonovi, F., Ferrara, A., & Piacentini, M. (2021). Performance decline in a low-stakes test at age 15 and educational attainment at age 25: Cross-country longitudinal evidence. Journal of Adolescence, 92, 114-125. https://doi.org/10.1016/j.adolescence.2021.08.011
8. Bridgeman, B., & Cline, F. (2000). Variations in mean response time for questions on the computer-adaptive GRE General Test: Implications for fair assessment. GRE Board Professional Report No. 96 20P. Educational Testing Service. https://doi.org/10.1002/j.2333-8504.2000.tb01830.x

9. Chae, Y.M., Park, S.G., & Park, I. (2019). The relationship between classical item characteristics and item response time on computer-based testing. Korean Journal of Medical Education, 31(1), 1-9. https://doi.org/10.3946/kjme.2019.113
10. Chen, G., Cheng, W., Chang, T.W., Zheng, X., & Huang, R. (2014). A comparison of reading comprehension across paper, computer screens, and tablets: Does tablet familiarity matter? Journal of Computers in Education, 1(3), 213 225. http://dx.doi.org/10.1007%2Fs40692-014-0012-z
11. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
12. Cole, J.S., Bergin, D.A., & Whittaker, T.A. (2008). Predicting student achievement for low stakes tests with effort and task value. Contemporary Educational Psychology, 33(4), 609–624. https://doi.org/10.1016/j.cedpsych.2007.10.002
13. Cooper, J. (2006). The digital divide: The special case of gender. Journal of Computer Assisted Learning, 22, 320–334. https://doi.org/10.1111/j.1365-2729.2006.00185.x
14. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Wadsworth.
15. Çokluk, Ö., Gül, E., & Doğan-Gül, C. (2016). Examining differential item functions of different item ordered test forms according to item difficulty levels. Educational Sciences-Theory & Practice, 16(1), 319-330. https://doi.org/10.12738/estp.2016.1.0329
16. DeMars, C.E. (2007). Changes in rapid-guessing behavior over a series of assessments. Educational Assessment, 12(1), 23–45. https://doi.org/10.1080/10627190709336946
17. Eklöf, H. (2007). Test-taking motivation and mathematics performance in the TIMSS 2003. International Journal of Testing, 7(3), 311 326. https://doi.org/10.1080/15305050701438074
18. Eklöf, H. (2010). Skill and will: Test-taking motivation and assessment quality. Assessment in Education Principles Policy Practice, 17, 345 356. https://doi.org/10.1080/0969594X.2010.516569
19. Fan, Z., Wang, C., Chang, H.-H., & Douglas, J. (2012). Response time distributions for item selection in CAT. Journal of Educational and Behavioral Statistics, 37(5), 655-670. http://dx.doi.org/10.3102/1076998611422912
20. Fishbein, B., Foy, P., & Yin, L. (2021). TIMSS 2019 user guide for the international database (2nd ed.). TIMSS & PIRLS International Study Center.
21. Gneezy, U., List, J.A., Livingston, J.A., Qin, X., Sadoff, S., & Xu, Y. (2019). Measuring success in education: the role of effort on the test itself. American Economic Review: Insights, 1(3), 291-308. http://dx.doi.org/10.1257/aeri.20180633
22. Guo, H., Rios, J.A., Haberman, S., Liu, O.L., Wang, J., & Paek, I. (2016). A new procedure for detection of students’ rapid guessing responses using response time. Applied Measurement in Education, 29(3), 173 183. https://doi.org/10.1080/08957347.2016.1171766
23. Hannula. (2004). Development of understanding and self-confidence in mathematics, grades 5-8. Proceding of the 28th Conference of the International Group for the Psychology of Mathematics Education, 3, 17-24. http://files.eric.ed.gov/fulltext/ED489565.pdf
24. Hess, B.J., Johnston, M.M., & Lipner, R.S. (2013). The impact of item format and examinee characteristics on response times. International Journal of Testing, 13(4), 295–313. https://doi.org/10.1080/15305058.2012.760098
25. Hoffman, B. (2010). “I think I can, but I'm afraid to try”: The role of self-efficacy beliefs and mathematics anxiety in mathematics problem-solving efficiency. Learning and Individual Differences, 20(3), 276-283. https://doi.org/10.1016/j.lindif.2010.02.001
26. Hoffman, B., & Spatariu, A. (2008). The influence of self-efficacy and metacognitive prompting on math problem-solving efficiency. Contemporary Educational Psychology, 33(4), 875-893. https://doi.org/10.1016/j.cedpsych.2007.07.002
27. İlgün-Dibek, M. (2020). Silent predictors of test disengagement in PIAAC 2012. Journal of Measurement and Evaluation in Education and Psychology, 11(4), 430-450. https://doi.org/10.21031/epod.796626
28. İlhan, M., Öztürk, N.B., & Şahin, M.G. (2020). The effect of the item’s type and cognitive level on its difficulty index: The sample of the TIMSS 2015. Participatory Educational Research, 7(2), 47-59. https://doi.org/10.17275/per.20.19.7.2
29. Koçdar, S., Karadağ, N., & Şahin, M.D. (2016). Analysis of the difficulty and discrimination indices of multiple-choice questions according to cognitive levels in an open and distance learning context. The Turkish Online Journal of Educational Technology, 15(4), 16–24. https://hdl.handle.net/11421/11442
30. Lasry, N., Watkins, J., Mazur, E., & Ibrahim, A. (2013). Response times to conceptual questions. American Journal of Physics, 81(9), 703 706. https://doi.org/10.1119/1.4812583
31. Lee, Y.H., & Chen, H. (2011). A review of recent response-time analyses in educational testing. Psychological Test and Assessment Modeling, 53(3), 359–379.
32. Lee, Y.H., & Jia, Y. (2014). Using response time to investigate students' test-taking behaviors in a NAEP computer-based study. Large-scale Assessments in Education, 2(8), 1-24. https://doi.org/10.1186/s40536-014-0008-1
33. Levine, T., & Donitsa-Schmidt, S. (1998). Computer use, confidence, attitudes, and knowledge: A causal analysis. Computers in Human Behavior, 14(1), 125 146. http://dx.doi.org/10.1016/0747-5632(93)90033-O
34. Lundgren, E., & Eklöf, H. (2020). Within-item response processes as indicators of test-taking effort and motivation. Educational Research and Evaluation, 26(5-6), 275-301. https://doi.org/10.1080/13803611.2021.1963940
35. Martin, M.O., von Davier, M., & Mullis, I.V.S. (Eds.). (2020). Methods and procedures: The TIMSS 2019 technical report. The TIMSS & PIRLS International Study Center. https://www.iea.nl/publications/technical-reports/methods-and-procedures-timss-2019-technical-report
36. Michaelides, M.P., Ivanova, M., & Nicolaou, C. (2020). The relationship between response-time effort and accuracy in PISA science multiple choice items. International Journal of Testing, 20(3), 187-205. https://doi.org/10.1080/15305058.2019.1706529
37. Ministry of National Education (2019, March 19). Muğla İl Millî Eğitim Müdürlüğü: The TIMSS 2019 [Muğla Provincial Directorate of National Education: TIMMS 2019] https://mugla.meb.gov.tr/www/timss-2019/icerik/2298
38. Momsen, J., Offerdahl, E., Kryjevskaia, M., Montplaisir, L., Anderson, E., & Grosz, N. (2013). Using assessments to investigate and compare the nature of learning in undergraduate science courses. CBE Life Sciences Education, 12(2), 239 249. https://doi.org/10.1187%2Fcbe.12-08-0130
39. Mullis, I.V.S., Martin, M.O., Goh, S., & Cotter, K. (Eds.). (2016). The TIMSS 2015 encyclopedia: Education policy and curriculum in mathematics and science. The TIMSS & PIRLS International Study Center. http://timssandpirls.bc.edu/timss2015/encyclopedia/
40. Mullis, I.V.S., & Martin, M.O. (2017). The TIMSS 2019 assessment frameworks. The TIMSS & PIRLS International Study Center. http://timssandpirls.bc.edu/timss2019/frameworks/
41. Myers, A.J., & Finney, S.J. (2021). Change in self-reported motivation before to after test completion: Relation with performance. The Journal of Experimental Education, 89, 74–94. https://doi.org/10.1080/00220973.2019.1680942
42. Nehm, R.H., & Schonfeld, M. (2008). Item feature effects in evolution assessment. Journal of Research in Science Teaching, 48(3), 237–256. https://doi.org/10.1002/tea.20400
43. Nevid, J.S., & McClelland, N. (2013). Using action verbs as learning outcomes: Applying Bloom’s taxonomy in measuring instructional objectives in introductory psychology. Journal of Education and Training Studies, 1(2), 19 24. http://dx.doi.org/10.11114/jets.v1i2.94
44. Organisation for Economic Co-operation and Development [OECD]. (2015). Using log-file data to understand what drives performance in PISA (case study), in students, computers and learning: Making the connection. OECD Publishing. https://doi.org/10.1787/9789264239555-en
45. Pommerich, M. (2004). Developing computerized versions of paper-and-pencil tests: Mode effects for passaged-based tests. Journal of Technology, Learning, and Assessment, 2(6), 1–45. https://files.eric.ed.gov/fulltext/EJ905028.pdf
46. Rabbani, S., & Herman, T. (2017). Increasing Formulate and Test Conjecture Math Competence and Self Confidence in Using the Discovery Learning Teaching Math. PrimaryEdu: Journal of Primary Education, 1(1), 119 128. http://dx.doi.org/10.22460/pej.v1i1.488
47. Rios, J.A., & Guo, H. (2020). Can culture be a salient predictor of test-taking engagement? An analysis of diferential nonefortful responding on an international college-level assessment of critical thinking. Applied Measurement in Education, 33(4), 263 279. http://dx.doi.org/10.1080/08957347.2020.1789141
48. Schnipke, D.L., & Scrams, D.J. (1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34, 213–232. https://psycnet.apa.org/doi/10.1111/j.1745-3984.1997.tb00516.x
49. Setzer, J.C., Wise, S.L., van de Heuvel, J.R., & Ling, G. (2013). An investigation of examinee test-taking effort on a large-scale assessment. Applied Measurement in Education, 26(1), 34–49. https://doi.org/10.1080/08957347.2013.739453
50. Silm, G., Pedaste, M., & Täht, K. (2020). The relationship between performance and test-taking effort when measured with self-report or time-based instruments: A meta-analytic review. Educational Research Review, 31, 100335. https://doi.org/10.1016/j.edurev.2020.100335
51. Sundre, D.L., & Kitsantas, A. (2004). An exploration of the psychology of the examinee: Can examinee self-regulation and test-taking motivation predict consequential and nonconsequential test performance?. Contemporary Educational Psychology, 29(1), 6-26. https://psycnet.apa.org/doi/10.1016/S0361-476X(02)00063-2
52. Swerdzewski, P.J., Harmes, J.C., & Finney, S.J. (2011). Two approaches for identifying low-motivated students in a low-stakes assessment context. Applied Measurement in Education, 24(2), 162–188. http://dx.doi.org/10.1080/08957347.2011.555217
53. Veeravagu, J., Muthusamy, C., Marimuthu, R., & Subrayan, A. (2010). Using Bloom’s taxonomy to gauge students’ reading comprehension performance. Canadian Social Science, 6(3), 205–212. https://doi.org/10.3968/J.CSS.1923669720100603.023
54. Walkington, C., Clinton, V., & Sparks, A. (2019). The effect of language modification of mathematics story problems on problem-solving in online homework. Instructional Science, 47(5), 499-529. https://link.springer.com/article/10.1007/s11251-019-09481-6
55. Wang, M. (2017). Characteristics of item response time for standardized achievement assessments [Doctoral dissertation]. University of Iowa.
56. Wang, T., & Hanson, B.A. (2005). Development and calibration of an item response model that incorporates response time. Applied Psychological Measurement, 29(5), 323-339. https://doi.org/10.1177/0146621605275984
57. Weirich, S., Hecht, M., Penk, C., Roppelt, A., & Böhme, K. (2017). Item position effects are moderated by changes in test-taking effort. Applied Psychological Measurement, 41(2), 115-129. https://doi.org/10.1177%2F0146621616676791
58. Wise, S.L. (2006). An investigation of the differential effort received by items on a low-stakes, computer based test. Applied Measurement in Education, 19(2), 95 114. https://doi.org/10.1207/s15324818ame1902_2
59. Wise, S.L. (2017). Rapid-guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice, 36(4), 52 61. https://doi.org/10.1111/emip.12165
60. Wise, S.L., & DeMars, C.E. (2005). Low examinee effort in low-stakes assessment: problems and potential solutions. Educational Assessment, 10(1), 1 17. https://doi.org/10.1207/s15326977ea1001_1
61. Wise, S.L., Kingsbury, G.G., Thomason, J., & Kong, X. (2004, April 13-15). An investigation of motivation filtering in a statewide achievement testing program. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.
62. Wise, S.L. & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer based tests. Applied Measurement in Education, 18(2), 163 183. https://doi.org/10.1207/s15324818ame1802_2
63. Wise, S.L., & Ma, L. (2012, April 13-17). Setting response time thresholds for a CAT item pool: The normative threshold method. In annual meeting of the National Council on Measurement in Education, Vancouver, Canada (pp. 163 183). https://www.nwea.org/resources/setting-response-time-thresholds-cat-item-pool-normative-threshold-method/
64. Wolgast, A., Schmidt, N., & Ranger, J. (2020). Test-taking motivation in education students: Task battery order affected within-test-taker effort and importance. Frontiers in Psychology, 11, 1–16. https://doi.org/10.3389/fpsyg.2020.559683
65. Yalçın, S. (2022). Examining students' item response times in eTIMSS according to their proficiency levels, selfconfidence, and item characteristics. Journal of Measurement and Evaluation in Education and Psychology, 13(1), 23 39. https://doi.org/10.21031/epod.999545
66. Yang, C.L., O Neill, T.R., & Kramer, G.A. (2002). Examining item difficulty and response time on perceptual ability test items. Journal of Applied Measurement, 3(3), 282-299.
67. Zenisky, A.L., & Baldwin, P. (2006). Using item response time data in test development and validation: Research with beginning computer users. Center for educational assessment report No, 593. Amherst, MA: University of Massachusetts, School of Education.
68. Zhao, W. (2020). Identification and validation of disengagement measures based on response time: An application to PISA 2012 digital math items [Master's thesis]. University of Oslo.
69. Zhang, T., Xie, Q., Park, B.J., Kim, Y.Y., Broer, M., & Bohrnstedt, G. (2016). Computer familiarity and its relationship to performance in three NAEP digital-based assessments (AIR-NAEP Working Paper No. 01-2016). American Institutes for Research.