Silent Predictors of Test Disengagement in PIAAC 2012

Abstract

Although the effects of test disengagement on the validity of the scores obtained from the data set have been examined in many studies, the predictors of the disengaged behaviors received relatively limited scholarly attention in low-stakes assessment, in particular, in international comparison studies. As such, the present study with a twofold purpose sets out to determine the best fitted explanatory item response theory model and examine the predictors of test disengagement. The data were collected by using items measuring literacy and numeracy skills of adults from different countries such as Norway, Austria, Ireland, France, Denmark, Germany, and Finland participated in PIAAC 2012. The results of the model with item and person characteristics demonstrated that adults tended to be disengaged on very difficult items. Similarly, age has a negative effect on test-taking engagement for adults in several countries such as France and Ireland, while several predictors such as educational attainment, readiness to learn, and the use of ICT skills at home and work had positive effects on test engagement. In addition, females exhibit a higher level of engagement in Norway. Overall, the findings suggested that the effect of the predictors on disengagement depended on the domain and country. So, this study brings further attention that the role of test disengagement should be a prerequisite practice before reaching a conclusion from international large-stake assessments.

Keywords

• explanatory item response theory model
• low-stakes assessment
• PIAAC
• test disengagement

References

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, & Joint Committee on Standards for Educational and Psychological Testing. (2014). Standards for educational and psychological testing. Washington, DC: AERA.
2. Asseburg, R., & Frey, A. (2013). Too hard, too easy, or just right? The relationship between effort or boredom and ability-difficulty fit. Psychological Test and Assessment Modeling, 55(1), 92–104.
3. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48
4. Bergdahl, N., Nouri, J., & Fors, U. (2019). Disengagement, engagement and digital skills in technology-enhanced learning. Education and Information Technologies, 149. doi:10.1007/s10639-019-09998-w.
5. Braun, H., Kirsch, I., Yamamoto, K., Park, J., & Eagan, M. K. (2011). An experimental study of the effects of monetary incentives on performance on the 12th-grade NAEP reading assessment. Teachers College Record, 113(11), 2309–2344.
6. Briggs, D. C. (2008). Using explanatory item response models to analyze group differences in science achievement. Applied Measurement in Education, 21(2), 89-118. doi: 10.1080/08957340801926086.
7. Bridgeman, B., & Cline, F. (2000). Variations in mean response time for questions on the computer-adaptive GRE General Test: Implications for fair assessment. GRE Board Professional Report No. 96-20P. Princeton, NJ: Educational Testing Service.
8. Brown, G. T. L., & Harris, L. R. (2016). Handbook of human and social conditions in assessment. New York, NY: Routledge.

9. de Ayala, R. J. (2009). The theory and practice of item response theory. New York: Guilford Press
10. Demars, C. E. (2007). Changes in rapid-guessing behavior over a series of assessments. Educational Assessment, 12(1), 23–45. doi:10.1080/10627190709336946
11. DeMars, C. E., Bashkov, B. M., & Socha, A. B. (2013). The role of gender in test-taking motivation under low-stakes conditions. Research and Practices in Assessment, 8, 69-82.
12. Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53(1), 109–132. doi:10.1146/annurev.psych.53.100901.135153.
13. Finn, B. (2015). Measuring motivation in low-stakes assessments. ETS Research Report Series, 2015(2),1–17. doi: 10.1002/ets2.12067
14. Goldhammer, F., Martens, T., & Lüdtke, O. (2017). Conditioning factors of test-taking engagement in PIAAC: an exploratory IRT modelling approach considering person and item characteristics. Large-scale Assessments in Education, 5(18), 1-25. doi: 10.1186/s40536-017-0051-9.
15. Goldhammer, F., Martens, T., Christoph, G., & Lüdtke, O. (2016). Test-taking engagement in PIAAC. Vol. 133. In: OECD Education Working Papers. Paris: OECD Publishing. Hox J. 2002. Multilevel analysis: Techniques and applications. Mahwah, NJ: Erlbaum.
16. Kiefer, T., Robitzsch, A., & Wu, M. (2016). TAM: Test analysis modules. R package version 1.99–6. Retrieved from http:// CRAN.R-project.org/package=TAM
17. Kong, X. J., Wise, S. L., & Bhola, D. S. (2007). Setting the response time threshold parameter to differentiate solution behavior from rapid-guessing behavior. Educational and Psychological Measurement, 67(4), 606–619.
18. Lee, Y.-H., & Jia, Y. (2014). Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study. Large-scale Assessments in Education, 2(1), 1–24. doi: 10.1186/s40536-014-0008-1.
19. Marrs, H., & Sigler, E. A. (2012). Male academic performance in college: The possible role of study strategies. Psychology of Men & Masculinity, 13(2), 227-241.
20. Masters, J., Schnipke, D. L., & Connor, C. (2005, April). Comparing item response times and difficulty for calculation items. Paper presented at the annual meeting of the American Educational Research Association, Montréal, Canada.
21. Mastuti, E., & Handoyo, S. (2017, October). Effects of individual differences on the performance in computer-based test (CBT). Paper presented at the 3rd ASEAN Conference on Psychology, Counselling, and Humanities (ACPCH). Malang, Indonesia.
22. Nagy, G., Nagengast, B., Becker, M., Rose, N., & Frey, A. (2018). Item position effects in a reading comprehension test: an IRT study of individual differences and individual correlates. Psychological Test and Assessment Modeling, 60(2), 165–187.
23. Organisation for Economic Co-operation and Development. (2013a). OECD skills outlook 2013: First results from the survey of adult skills. Paris: OECD Publishing.
24. Organisation for Economic Co-operation and Development. (2013b), “The methodology of the Survey of Adult Skills (PIAAC) and the quality of data”, in The Survey of Adult Skills: Reader's Companion, OECD Publishing, Paris.
25. Organisation for Economic Co-operation and Development. (2013c). What the Survey of Adult Skills (PIAAC) measures in The Survey of Adult Skills: Reader's Companion, OECD Publishing, Paris. doi: https://doi.org/10.1787/9789264204027-4-en
26. Organisation for Economic Co-operation and Development. (2015). Adults, Computers and Problem Solving: What's the Problem?, OECD Skills Studies, OECD Publishing, Paris, https://doi.org/10.1787/9789264236844-en.
27. Organisation for Economic Co-operation and Development. (2016a).Technical report of the Survey of Adult Skills (PIAAC) (2nd edition). OECD, Paris,
28. Organisation for Economic Co-operation and Development. (2016b). The Survey of Adult Skills: Reader’s Companion, Second Edition, OECD Skills Studies, OECD Publishing, Paris. doi: 10.1787/9789264258075-en.
29. Organisation for Economic Co-operation and Development. (2017). Programme for the International Assessment of Adult Competencies (PIAAC), Log Files, GESIS Data Archive, Cologne, doi:10.4232/1.12955.
30. Organisation for Economic Co-operation and Development. (2019), Beyond Proficiency: Using Log Files to Understand Respondent Behaviour in the Survey of Adult Skills, OECD Skills Studies, OECD Publishing, Paris. doi: 10.1787/0b1414ed-en.
31. Perry, A., Helmschrott, S., Konradt, I., & Maehler, D. B. (2017). User Guide for the German PIAAC Scientific Use File: Version II. (GESIS Papers, 2017/23). Köln: GESIS - Leibniz-Institut für Sozialwissenschaften. https://nbn-resolving.org/urn:nbn:de:0168-ssoar-54438-v2-7
32. Rios, J. A., & Guo, H. (2020). Can culture be a salient predictor of test-taking engagement? An analysis of differential noneffortful responding on an international college-level assessment of critical thinking. Applied Measurement in Education. Advance online publication. doi: 10.1080/08957347.2020.1789141
33. Rios, J. A., Guo, H., Mao, L., & Liu, O. L. (2017). Evaluating the impact of careless responding on aggregated scores: To filter unmotivated examinees or not? International Journal of Testing, 17(1),74-104. doi: 10.1080/15305058.2016.1231193
34. Smith, M C., Rose, A.D., Smith, T. J.& Ross-Gordon, J. M. (2015, May). Adults’ readiness to learn and skill acquisition and use: An analysis of PIAAC. Paper presented at the 56th Annual Adult Education Research Conference. Manhattan, KS.
35. Schnipke, D. L., & Scrams, D. J. (2002). Exploring issues of examinee behavior: Insights gained from response-time analyses. In C. N. Mills, M. T. Potenza, J. J. Fremer, & W. C. Ward (Eds.), Computer-based testing: Building the foundation for future assessments (pp. 237-266). Mahwah, NJ: Lawrence Erlbaum Associates.
36. Setzer, J. C., Wise, S. L., van den Heuvel, J. R., & Ling, G. (2013). An investigation of examinee test-taking effort on a largescale assessment. Applied Measurement in Education, 26(1), 34–49. doi: 10.1080/08957347.2013.739453
37. Sundre, D. L., & Kitsantas, A. (2004). An exploration of the psychology of the examinee: Can examinee self-regulation and test-taking motivation predict consequential and non-consequential test performance? Contemporary Educational Psychology, 29(1), 6–26. doi:10.1016/S0361-476X(02)00063-2.
38. Team, R. C. (2016). R: A language and environment for statistical computing (Version 3.1.3). Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/.
39. Xie, B. (2003). Older adults, computers, and the Internet: Future directions. Gerontechnology,2(4), 289-305.
40. van Barnevald, C. (2007). The effect of examinee motivation on test construction within an IRT framework. Applied Psychological Measurement, 31(1), 31–46. doi:10.1177/0146621606286206
41. Wise, S. L. (2006). An investigation of the differential effort received by items on a low-stakes, computer-based test. Applied Measurement in Education, 19(2), 25-114.
42. Wise, S. L. (2009). Strategies for managing the problem of unmotivated examinees in low-stakes testing programs. The Journal of General Education, 58(3), 152–166. doi:10.1353/jge.0.0042
43. Wise, S. L. (2015). Effort analysis: Individual score validation of achievement test data. Applied Measurement in Education, 28(3), 237–252. doi:10.1080/08957347.2015.1042155
44. Wise, S. L. , & DeMars, C. E. (2009). A Clarification of the effects of rapid guessing on Coefficient α: A note on Attali’s “reliability of speeded number-right multiple-choice tests”. Applied Psychological Measurement , 33(6), 488–490. doi:10.1177/0146621607304655
45. Wise S. L. & DeMars C. E. (2010). Examinee noneffort and the validity of program assessment results. Educational Assessment, 15(1), 27-41.
46. Wise, S. L., & Kingsbury, G. G. (2015). Modeling student test-taking motivation in the context of an adaptive achievement test. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
47. Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. doi:10.1207/s15324818ame1802_2
48. Yang, C. L., O’Neill, T. R., & Kramer, G. A. (2002). Examining item difficulty and response time on perceptual ability test items. Journal of Applied Measurement, 3(3), 282-299. Zikmund, W.G., Babin, B.J., Carr, J.C. and Griffin, M. (2010). Business research methods, Canada:South-Western Cengage Learning.