Examining Students' Formative Test-Taking Behaviors Using Learning Analytics

Abstract

In online learning environments, assessment is an important dimension and also one of the most challenging parts of the process. So to provide an effective learning process analyzing students’ behaviors are important for designing formative and summative assessment environments. In this study, students’ profiles have been analyzed in a formative assessment environment and compared with the summative assessment environment based on attempt count, overall time spent, first attempt score, and the last attempt score metrics. The within-subjects design has been used. Cluster analysis and the Kruskal Wallis-H Test were used to analyze behaviors. The data show that there are three main clusters. Cluster 1 showed a high number of interactions, and an increasing trend was observed in grades over attempts. Cluster 2 consists of the students who received the best grades in all of the other clusters, and lastly, cluster 3 consists of students that interact little and score lower on formative assessments.

Keywords

• test-taking behavior analysis
• learning analytics
• formative assessment
• assessment analytics

References

1. Blondeel, E., Everaert, P., & Opdecam, E. (2023). Does practice make perfect? The effect of online formative assessments on students’ self-efficacy and test anxiety. The British Accounting Review (101189). https://doi.org/10.1016/j.bar.2023.101189
2. Cassady, J. C., & Gridley, B. E. (2005). The Effects of Online Formative and Summative Assessment on Test Anxiety and Performance. The Journal of Technology, Learning and Assessment, 4(1). Retrieved from https://ejournals.bc.edu/index.php/jtla/article/view/1648
3. Crisp, V., & Ward, C. (2008). The development of a formative scenario-based computer assisted assessment tool in psychology for teachers: The PePCAA project. Computers & Education, 50(4), 1509-1526. https://doi.org/10.1016/j.compedu.2007.02.004
4. Gašević, D., Greiff, S., & Shaffer, D. W. (2022). Towards strengthening links between learning analytics and assessment: Challenges and potentials of a promising new bond. Computers in Human Behavior, 134(2022), 107304. https://doi.org/10.1016/j.chb.2022.107304
5. Guo, H. (2021). How Did Students Engage with a Remote Educational Assessment? A Case Study. Educational Measurement: Issues and Practice, 41(3), 58-68. https://doi.org/10.1111/emip.12476
6. Guo, H., & Ercikan, K. (2021). Comparing Test‐Taking Behaviors of English Language Learners (ELLs) to Non‐ELL Students: Use of Response Time in Measurement Comparability Research. ETS Research Report Series, 2021(1), 1-15. https://doi.org/10.1002/ets2.12340
7. Han, S., & Kang, H.-A. (2021). Sequential Monitoring of Aberrant Test-Taking Behaviors Based on Response Times. In M. Wiberg, D. Molenaar, J. González, U. Böckenholt, & J.-S. Kim, Quantitative Psychology Cham. https://doi.org/10.1007/978-3-030-74772-5_7
8. Harlen, W., & James, M. (1997). Assessment and Learning: differences and relationships between formative and summative assessment. Assessment in Education: Principles, Policy & Practice, 4(3), 365-379. https://doi.org/10.1080/0969594970040304

9. Hui, B. (2023). Are They Learning or Guessing? Investigating Trial-and-Error Behavior with Limited Test Attempts. Paper presented at the LAK23: 13th International Learning Analytics and Knowledge Conference, Arlington, TX, USA. https://doi.org/10.1145/3576050.3576068
10. Lee, Y. H., & Haberman, S. J. (2016). Investigating Test-Taking Behaviors Using Timing and Process Data. International Journal of Testing, 16(3), 240-267. https://doi.org/10.1080/15305058.2015.1085385
11. Liao, M., Patton, J., Yan, R., & Jiao, H. (2021). Mining Process Data to Detect Aberrant Test Takers. Measurement: Interdisciplinary Research and Perspectives, 19(2), 93-105. https://doi.org/10.1080/15366367.2020.1827203
12. Lundgren, E., & Eklöf, H. (2020). Within-item response processes as indicators of test-taking effort and motivation. Educational Research and Evaluation, 26(5-6), 275-301. https://doi.org/10.1080/13803611.2021.1963940
13. Man, K., Harring, J. R., Ouyang, Y., & Thomas, S. L. (2018). Response Time Based Nonparametric Kullback-Leibler Divergence Measure for Detecting Aberrant Test-Taking Behavior. International Journal of Testing, 18(2), 155-177. https://doi.org/10.1080/15305058.2018.1429446
14. Palmen, L. N., Vorstenbosch, M. A. T. M., Tanck, E., & Kooloos, J. G. M. (2015). What is more effective: a daily or a weekly formative test? Perspectives on Medical Education, 4(2), 73-78. https://doi.org/10.1007/s40037-015-0178-8
15. Pools, E., & Monseur, C. (2021). Student test-taking effort in low-stakes assessments: evidence from the English version of the PISA 2015 science test. Large-scale Assessments in Education, 9(1), 10. https://doi.org/10.1186/s40536-021-00104-6
16. Rakoczy, K., Pinger, P., Hochweber, J., Klieme, E., Schütze, B., & Besser, M. (2019). Formative assessment in mathematics: Mediated by feedback's perceived usefulness and students' self-efficacy. Learning and Instruction, 60(2019), 154-165. https://doi.org/10.1016/j.learninstruc.2018.01.004
17. Rienties, B., & Toetenel, L. (2016). The impact of learning design on student behaviour, satisfaction and performance: A cross-institutional comparison across 151 modules. Computers in Human Behavior, 60(2016), 333–341. https://doi.org/10.1016/j.chb.2016.02.074
18. Sarac, M., & Loken, E. (2023). Examining patterns of omitted responses in a large-scale English language proficiency test. International Journal of Testing, 23(1), 56-72. https://doi.org/10.1080/15305058.2022.2070756
19. Shin, J., Chen, F., Lu, C., & Bulut, O. (2022). Analyzing students’ performance in computerized formative assessments to optimize teachers’ test administration decisions using deep learning frameworks. Journal of Computers in Education, 9(1), 71-91. https://doi.org/10.1007/s40692-021-00196-7
20. Silm, G., Must, O., & Täht, K. (2013). Test-taking effort as a predictor of performance in low-stakes tests. Trames. Journal of the Humanities and Social Sciences, 17(67/62), 433-448. https://doi.org/10.3176/tr.2013.4.08
21. Silm, G., Pedaste, M., & Täht, K. (2020). The relationship between performance and test-taking effort when measured with self-report or time-based instruments: A meta-analytic review. Educational Research Review, 31(100335), 1-22. https://doi.org/10.1016/j.edurev.2020.100335
22. Stadler, M., Hofer, S., & Greiff, S. (2020). First among equals: Log data indicates ability differences despite equal scores. Computers in Human Behavior, 111(2020), 106442. https://doi.org/10.1016/j.chb.2020.106442
23. Stenlund, T., Eklof, H., & Lyren, P. E. (2017). Group differences in test-taking behaviour: an example from a high-stakes testing program. Assessment in Education-Principles Policy & Practice, 24(1), 4-20. https://doi.org/10.1080/0969594x.2016.1142935
24. Stenlund, T., Lyren, P. E., & Eklof, H. (2018). The successful test taker: exploring test-taking behavior profiles through cluster analysis. European Journal of Psychology of Education, 33(2), 403-417. https://doi.org/10.1007/s10212-017-0332-2
25. Tempelaar, D., Rienties, B., Mittelmeier, J., & Nguyen, Q. (2018). Student profiling in a dispositional learning analytics application using formative assessment. Computers in Human Behavior, 78(2018), 408-420. https://doi.org/10.1016/j.chb.2017.08.010
26. Toetenel, L., & Rienties, B. (2016). Learning Design – creative design to visualise learning activities. Open Learning: The Journal of Open, Distance and e-Learning, 31(3), 233-244. https://doi.org/10.1080/02680513.2016.1213626
27. Vonderwell, S., Liang, X., & Alderman, K. (2007). Asynchronous Discussions and Assessment in Online Learning. Journal of Research on Technology in Education, 39(3), 309-328. https://doi.org/10.1080/15391523.2007.10782485
28. Wise, S. L. (2006). An Investigation of the Differential Effort Received by Items on a Low-Stakes Computer-Based Test. Applied Measurement in Education, 19(2), 95-114. https://doi.org/10.1207/s15324818ame1902_2
29. Wise, S. L., & DeMars, C. E. (2006). An Application of Item Response Time: The Effort-Moderated IRT Model. Journal of Educational Measurement, 43(1), 19-38. https://doi.org/https://doi.org/10.1111/j.1745-3984.2006.00002.x
30. Wise, S. L., Ma, L., Cronin, J., & Theaker, R. A. (2013). Student test-taking effort and the assessment of student growth in evaluating teacher effectiveness. Annual conference of the American Educational Research Association, San Francisco, CA,
31. Wise, S. L., & Gao, L. (2017). A General Approach to Measuring Test-Taking Effort on Computer-Based Tests. Applied Measurement in Education, 30(4), 343-354. https://doi.org/10.1080/08957347.2017.1353992
32. Wolsey, T. (2008). Efficacy of Instructor Feedback on Written Work in an Online Program. EdMedia + Innovate Learning Online 2022, 7(2), 311-329.
33. Xiong, Y., & Suen, H. K. (2018). Assessment approaches in massive open online courses: Possibilities, challenges, and future directions. International Review of Education, 64(2), 241-263. https://doi.org/10.1007/s11159-018-9710-5
34. Yang, A. C. M., Chen, I. Y. L., Flanagan, B., & Ogata, H. (2022). How students’ self-assessment behavior affects their online learning performance. Computers and Education: Artificial Intelligence, 3(2022), 100058. https://doi.org/10.1016/j.caeai.2022.100058
35. Yildirim-Erbasli, S. N., & Bulut, O. (2020). The impact of students’ test-taking effort on growth estimates in low-stakes educational assessments. Educational Research and Evaluation, 26(7-8), 368-386. https://doi.org/10.1080/13803611.2021.1977152
36. Yildirim-Erbasli, S. N., & Bulut, O. (2022). Designing Predictive Models for Early Prediction of Students’ Test-taking Engagement in Computerized Formative Assessments. Journal Of Applied Testing Technology, 22(2), 1-14. Retrieved from https://jattjournal.net/index.php/atp/article/view/167548