Identifying the Careless Responders: A Cross-Country Comparison on PISA 2018 Dataset
Öz
Careless responding (CR) is one of the major concerns in self-report measures since it affects the quality of collected data. In this research, it was aimed to examine the dataset in terms of CR, to investigate the effects of CR on the psychometric properties of data, and to examine the effectiveness of CR detection methods in a cross-cultural context. Specifically; response time, outlier and long-string indices were used to flag CR and efficiency of using these were compared on PISA 2018 reading related attitude scales for Singapore and Turkey. As a result, it was indicated that the amount of CR was higher for Singapore than Turkey at long-string and response time methods. Furthermore, after removal of CR from dataset, enjoyment level of reading, perception of competence and perception of difficulty of PISA tests scores increased. Another result was that long-string and response time analyses provided significant improvement in model fit and long-string had the highest improvement level. Moreover, removing respondents according to outlier analyses didn’t provide any improvement on model-data fit. In general, results illustrated that careless responding behavior may have important effects on psychometric properties of self-report and screening the data for CR is strongly suggested in future studies.
Anahtar Kelimeler
Careless Responding, Insufficient Effort Responding, Unmotivated Responding, PISA 2018, Self-report Data
Kaynakça
- Alarcon, G. M., & Lee, M. A. (2022). The relationship of insufficient effort responding and response styles: An online experiment. Frontiers in Psychology, 12. https://www.frontiersin.org/article/10.3389/fpsyg.2021.784375
- Beach, D. A. (1989) Identifying the random responder. The Journal of Psychology, 123(1), 101-103, DOI: 10.1080/00223980.1989.10542966
- Beck, M. F., Albano, A. D., & Smith, W. M. (2019). Person-fit as an index of inattentive responding: A comparison of methods using polytomous survey Data. Applied Psychological Measurement, 43(5), 374–387. https://doi.org/10.1177/0146621618798666
- Bowling, N. A., Huang, J. L., Brower, C. K., & Bragg, C. B. (2021). The quick and the careless: the construct validity of page time as a measure of insufficient effort responding to surveys. Organizational Research Methods. https://doi.org/10.1177/10944281211056520
- Brühlmann, F., Petralito, S., Aeschbach, L. F., & Opwis, K. (2020). The quality of data collected online: An investigation of careless responding in a crowdsourced sample. Methods in Psychology, 2. https://doi.org/10.1016/j.metip.2020.100022
- Curran, P. G. (2015). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66(2016), 4–19. https://doi.org/10.1016/j.jesp.2015.07.006
- Eklöf, H., Pavešič, B. J., & Grønmo, L. S. (2014). A cross-national comparison of reported effort and mathematics performance in TIMSS Advanced. Applied Measurement in Education, 27(1), 31–45. https://doi.org/10.1080/08957347.2013.853070
- Emons, W. H. (2008). Nonparametric person-fit analysis of polytomous item scores. Applied Psychological Measurement, 32 (3), 224–247. https://doi.org/10.1177/0146621607302479
- Goldammer, P., Annen, H., Stöckli, P. L., & Jonas, K. (2020). Careless responding in questionnaire measures: Detection, impact, and remedies. The Leadership Quarterly, 31(4). https://doi.org/10.1016/j.leaqua.2020.101384
- Goldhammer, F., Martens, T., Christoph, G., & Lüdtke, O. (2016). Test-taking engagement in PIAAC. Vol. 133. In: OECD Education Working Papers. Paris: OECD Publishing.