Detecting Careless Responses to Self-Reported Questionnaires

Abstract

Problem Statement: The use of self-report questionnaires may lead to biases such as careless responses that distort the research outcomes. Early detection of careless responses in self-report questionnaires may reduce error, but little guidance exists in the literature regarding techniques for detecting such careless or random responses in self-report questionnaires. Purpose of the Study: The purpose of this study was to examine whether the respondent‟s goodness-of-fit test score (RGF) can be used to indicate careless responses in completing self-report questionnaires. It is hypothesized that there is a significant difference of RGF between careless responses and true responses and that RGF of careless responses is higher than RGF for true responses. Method: An experimental research design that made use of a self-reported questionnaire was conducted with 205 respondents divided into two groups. The first group responded truthfully to the questionnaire while the second group responded carelessly to the questionnaire. The validity and reliability of the questionnaire had been tested. One hundred and eighty five respondents were selected as the group of true responses, while another 20 respondents comprised the group of careless responses. T-test of independent sample was used to evaluate the different RGF among true responses and careless responses. Findings: After comparing the mean scores of RGF between careless responses and true responses, a significant difference was found. The frequency distribution of true responses tends to be normally distributed while the existence of careless responses creates a skewed distribution to the right. The RGF of careless responses is higher than the RGF of true responses. Conclusion and Recommendations: RGF may be used as an indicator of respondent‟s careless responses in self-report questionnaires in which more accurate data are expected. Social science research that makes use of self-report questionnaire in measuring affective domain may compute RGF to determine whether careless responses exist. Keywords: Careless response, Questionnaire development, Random response, Goodness-of-fit.

References

1. Begin, G., Boivin, M. & Bellerose, J. (1979). Sensitive data collection through the
2. random response technique: Some improvements. Journal of Psychology.
3. (1), 53-65.
4. Castro, R. (2013). Inconsistent respondents and sensitive questions. Field Methods.
5. (3), 283-298. doi: 10.1177/1525822X12466988
6. Conijn, J.M., Emons, W.H.M., Van Assen, M.A.L.M., Pedersen, S.S., & Sijtsma, K.
7. (2013). Explanatory, multilevel person-fit analysis of response consistency on
8. the Spielberger state-trait anxiety inventory. Multivariate Behavioral Research,

9. (5), 692-718.
10. Crede, M. (2010). Random responding as a threat to the validity of effect size
11. estimates in correlational research. Educational and Psychological Measurement,
12. (4), 596-612.
13. Crino, M.D. (1985). The random response technique as an indicator of questionnaire
14. item social desirability/personal sensitivity. Educational and Psychological
15. Measurement, 45(3), 453-468.
16. Godinho, A., Kushnir, V., & Cunningham, J.A. (2016). Unfaithful findings:
17. Identifying careless responding in addictions research. Addiction, 111(6), 955-
18. Keeley, J.W., Webb, C., Peterson, D., Roussin, L., & Flanagan, E.H. (2016).
19. Development of a response inconsistency scale for the personality inventory
20. for DSM-5. Journal of Personality Assessment, 98(4), 351-359.
21. Escobal, J., & Benites, S. (2013). PDAs in socio-economic surveys: instrument bias,
22. surveyor bias or both? International Journal of Social Research Methodology,
23. (1), 47-63. doi: 10.1080/13645579.2011.648420
24. Garcia, A.A. (2011). Cognitive interviews to test and refine questionnaires. Public
25. Health Nursing, 28(5), 444-450. doi: 10.1111/j.1525-1446.2010.00938.x
26. Johnson, T. P., & Wislar, J. S. (2012). Response rates and nonresponse errors in
27. surveys. Journal of the American Medical Association, 307(17), 1805-1806.
28. Keller, G. & Warrack, B. (2016). Statistics for management and economics, 9th ed.
29. Australia: Thomson.
30. Lara, D., Garcia, S.G., Ellertson, C., Camlin, C. & Suarez, J. (2006). The measure of
31. induced abortion levels in Mexico using random response technique.
32. Sociaological Method & Research, 35(2), 279-301.
33. Meade, A.W., & Craig, S.B. (2012). Identifying careless responses in survey data.
34. Psychological Methods, 17(3), 437-455.
35. Meyer, J.F., Fraust, K.A., Faust, D., Baker, A.M., & Cook, N.E. (2013). Careless and
36. random responding on clinical and research measures in the addictions: A
37. concerning problem and investigation of their detection. International Journal
38. of Mental Health and Addiction, 7(3), 292-306.
39. Penwarden, R. (2013, August). How to avoid nonresponse bias. FluidSurveys.
40. Retrieved from http://fluidsurveys.com/how-to-avoid-nonresponse-bias.
41. Reaser, J.M. (1975). A test of the forced-alternative random response questionnaire
42. technique. Technical Report no. 75-9.
43. Rogelberg, S.G., Fisher, G.G., Maynard, D.C., Hakel, M.D., & Horvath, M. (2001).
44. Attitudes toward surveys: Development of a measure and its relationship to
45. respondent behavior. Organizational Research Methods, 4(1), 3-25.
46. Rousseau, B., & Ennis, J.M. (2013). Importance of correct instructions in the Tetrad
47. test. Journal of Sensory Studies, 28(4), 264-269. doi: 10.1111/joss.12049
48. Spector, Paul E. (1992). Summated rating scale construction: An introduction.
49. California: Sage Publications, Inc. p. 30-31.
50. Sriramatr, S., Berry, T.R., Rodgers, W., & Stolp, S. (2012). The effect of different
51. response formats on ratings of exerciser stereotypes. Social Behavior and
52. Personality, 40(10), 1655-1666.
53. Summers, G.F., & Hammonds, A.D. (1969). Toward a paradigm for respondent bias
54. in survey research. Sociological Quarterly, 10(1), 113-121.
55. Tabachnick, B. G. & Fidel L. S. (2013). Using multivariate statistics, 6th ed. Boston:
56. Pearson, p. 661.
57. Tatsuoka & Tatsuoka (as cited in Harnisch, 1981). Analysis of item response
58. patterns: Consistency indices and their application to criterion-reference tests.
59. Paper presented at the Annual Meeting of the American Educational Research
60. Association, Long Angeles, April 13-17, 1981.
61. Thompson, B., Melancon, J.G., & Kier, F.J. (1998). Faking/random response scales for
62. the PPSDQ-93 measure of Jungian personality types. Paper presented at the
63. annual meeting of the Southwestern Psychological Association (new Orleans, L.A.,
64. April).
65. Warner, S. L. (as cited in Begin, Boivin & Bellerose, 1979). Sensitive data collection
66. through the random response technique: Some improvements. Journal of
67. Psychology, 101(1), 53-65.