The effects of reverse items on psychometric properties and respondents’ scale scores according to different item reversal strategies

Öz

This study aimed to examine the effects of reverse items created with different strategies on psychometric properties and respondents’ scale scores. To this end, three versions of a 10-item scale in the research were developed: 10 positive items were integrated in the first form (Form-P) and five positive and five reverse items in the other two forms. The reverse items in the second and third forms were crafted using antonyms (Form-RA) and negations (Form-RN), respectively. Based on the research results, Form-P was unidimensional, while other forms were two-dimensional. Moreover, although reliability coefficients of all forms were obtained as above .80, the lowest one was acquired for Form-RN. There were strong-positive relationships between students’ scores in the three scale forms. However, the lowest one was estimated between Form-P and Form-RN. Finally, there was a significant difference between the students’ mean scores obtained from Form–RN and other two versions, but the effect size of the said difference was small. In conclusion, all these results indicate that different types of reverse items influence psychometric properties and respondents’ scale scores differently.

Anahtar Kelimeler

• Antonyms
• Negations
• Item reversal strategy
• Item wording
• Reverse worded items

References

1. Ahlawat, K.S. (1985). On the negative valence items in self-report measures. The Journal of General Psychology, 112(1), 89–99. https://doi.org/10.1080/00221309.1985.9710992
2. Aybek, E.C. (2021). Data preparation for factor analysis. URL: https://shiny.eptlab.com/dp2fa/
3. Bandalos, D.L. (2018). Measurement theory and applications for the social sciences. The Guilford.
4. Benson, J., & Hocevar, D. (1985). The impact of item phrasing on the validity of attitude scales for elementary school children. Journal of Educational Measurement, 22(3), 231–240. https://doi.org/10.1111/j.1745-3984.1985.tb01061.x
5. Bergstrom, B.A., & Lunz, M.E. (1998, 13–17, April). Rating scale analysis: Gauging the impact of positively and negatively worded items. Annual Meeting of the American Educational Research Association, San Diego.
6. Boley, B.B., Jordan, E., & Woosnam, K.M. (2021). Reversed polarity items in tourism scales: Best practice or dimensional pitfall? Current Issues in Tourism, 24(4), 1 13. https://doi.org/10.1080/13683500.2020.1774517
7. Bolt, D., Wang, Y.C., Meyer, R.H., & Pier, L. (2020). An IRT mixture model for rating scale confusion associated with negatively worded items in measures of social-emotional learning. Applied Measurement in Education, 33(4), 1 18. https://doi.org/10.1080/08957347.2020.1789140
8. Brace, I., & Bolton, I. (2022). Questionnaire design: How to plan, structure and write survey material for effective market research (5th ed.). Kogan Page.

9. Bulut, H.Ç. (2021). Item wording effects in psychological measures: Do early literacy skills matter? Journal of Measurement and Evaluation in Education and Psychology, 12(3), 239-253. https://doi.org/10.21031/epod.944067
10. Bulut, H.Ç., & Bulut, O. (2022). Item wording effects in self-report measures and reading achievement: Does removing careless respondents help? Studies in Educational Evaluation, 72(2). https://doi.org/10.1016/j.stueduc.2022.101126
11. Carlson, M., Wilcox, R., Chou, C.-P., Chang, M., Yang, F., Blanchard, J., Marterella, A., Kuo, A., & Clark, F. (2011). Psychometric properties of reverse-scored items on the CES-D in a sample of ethnically diverse older adults. Psychological Assessment, 23(2), 558–562. https://doi.org/10.1037/a0022484
12. Chamberlain, V.M., & Cummings, M.N. (1984). Development of an instructor/course evaluation instrument. College Student Journal, 18(3), 246–250.
13. Chyung, S.Y.Y., Barkin, J.R., & Shamsy, J.A. (2018). Evidence-based survey design: The use of negatively worded items in surveys. Performance Improvement, 57(3), 16–25. https://doi.org/10.1002/pfi.21749
14. Coleman, C.M. (2013). Effects of negative keying and wording in attitude measures: A mixed-methods study [Unpublished doctoral dissertation, James Madison University]. https://commons.lib.jmu.edu/diss201019/73/
15. Conrad, K.J., Wright, B.D., McKnight P., McFall, M., Fontana, A., & Rosenheck, R. (2004). Comparing traditional and Rasch analyses of the Mississippi PTSD Scale: Revealing limitations of reverse-scored items. Journal of Applied Measurement, 5(1), 15–30.
16. Coolican, H. (2013). Research methods and statistics in psychology (5th ed.). Routledge.
17. Dagnew, A. (2017). The relationship between students’ attitudes towards school, values of education, achievement motivation and academic achievement in Gondar secondary schools, Ethiopia. Research in Pedagogy, 7(1), 30–42. https://doi.org/10.17810/2015.46
18. Dawson, J. (2017). Analysing quantitative survey data for business and management students. Sage.
19. DeVellis, R. (2017). Scale development: Theory and applications (4th ed.). Sage.
20. Diedenhofen, B., & Musch, J. (2015). Cocor: A comprehensive solution for the statistical comparison of correlations. PLoSONE, 10(4), 1 12. https://doi.org/10.1371/journal.pone.0121945
21. Dooden, H. (2014). The effects of positively and negatively worded items on the factor structure of the UCLA loneliness scale. Journal of Psychoeducational Assessment, 33(3), 259–267. https://doi.org/10.1177/0734282914548325
22. Dueber, D.M., Toland, M.D., Lingat, J.E., Love, A.M.A., Qiu, C., Wu, R., & Brown, A.V. (2021). To Reverse item orientation or not to reverse item orientation, that is the question. Assessment, 29(7), 1422–1440. https://doi.org/10.1177/10731911211017635
23. Dunbar, M., Ford, G., Hunt, K., & Der, G. (2000). Question wording effects in the assessment of global self-esteem. European Journal of Psychological Assessment, 16(1), 13–19. https://doi.org/10.1027//1015-5759.16.1.13
24. Ebesutani, C., Drescher, C.F., Reise, S.P., Heiden, L., High, T.L., Damon, J.D., & Young, J. (2012). The loneliness questionnaire-short version: An evaluation of reverse- worded and non-reverse-worded items via item response theory. Journal of Personality Assessment, 94(4), 427–437. https://doi.org/10.1080/00223891.2012.662188
25. Fabrigar, L.R., Wegener, D.T., MacCallum, R.C., & Strahan, E.J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299. http://dx.doi.org/10.1037/1082-989X.4.3.272
26. Finney, S.J. (2001). A comparison of the psychometric properties of negatively and positively worded questionnaire items (Publication No. 3009716) [Doctoral dissertation, University of Nebraska]. ProQuest Dissertations & Theses Global.
27. Furr, R.M. (2018). Psychometrics: An introduction (3rd ed.). Sage.
28. Geddes, J.D., Murrell, A.R., & Bauguss, J. (2010). Childhood learning: An examination of ability and attitudes toward school. Creative Education, 1(3), 170 183. https://doi.org/10.4236/ce.2010.13027
29. Gnambs, T., & Schroeders, U. (2020). Cognitive abilities explain wording effects in the Rosenberg Self Esteem Scale. Assessment, 27(2), 404 418. https://doi.org/10.1177/1073191117746503
30. Greenberger, E., Chen, C., Dmitrieva, J., & Farruggia, S.P. (2003). Item-wording and the dimensionality of the Rosenberg Self-Esteem Scale: Do they matter? Personality and Individual Differences, 35(6), 1241 1254. https://doi.org/10.1016/S0191 8869(02)00331-8
31. Guyatt, G.H., Cook, D.J., King, D., Norman, G.R., Kane, S.L., & Van Ineveld, C. (1999). Effect of the framing of questionnaire items regarding satisfaction with training on residents’ responses. Academic Medicine, 74(2), 192 194. https://doi.org/10.1097/00001888 199902000-00018
32. Hazlett-Stevens, H., Ullman, J.B., & Craske, M.G. (2004). Factor structure of the Penn State worry questionnaire: Examination of a method factor. Assessment, 11(4), 361–370. https://doi.org/10.1177/1073191104269872
33. Herche, J., & England, B. (1996). Reversed-polarity items and scale unidimensionality. Journal of the Academy of Marketing Science, 24(4), 366 374. https://doi.org/10.1177/0092070396244007
34. Hooper, M., Arora, A., Martin, M.O., & Mullis, I.V.S, (2013, 24–25 June). Examining the behavior of “reverse directional” items in the TIMSS 2011 context questionnaire scales [Conference presentation]. 5th IEA International Research Conference. National Institute of Education, Nanyang Technological University, Singapore. https://www.iea.nl/sites/default/files/2019-04/IRC-2013_Hooper_etal.pdf
35. Hughes, G.D. (2009). The impact of incorrect responses to reverse-coded survey items. Research in the Schools, 16(2), 76–88.
36. Ilhan, M., & Guler, N. (2017). The number of response categories and the reverse scored item problem in Likert-type scales: A study with the Rasch model. Journal of Measurement and Evaluation in Education and Psychology, 8(3), 321 343. http://dx.doi.org/10.21031/epod.321057
37. JASP Team (2022). JASP (Version 0.16.3) [Computer software].
38. Johnson, J.M., Bristow, D.N., & Schneider, K.C. (2004). Did you not understand the question or not? An investigation of negatively worded questions in survey research. Journal of Applied Business Research, 20(1), 75–86. https://doi.org/10.19030/jabr.v20i1.2197
39. Kam, C.C.S. (2023). Why do regular and reversed items load on separate factors? Response difficulty vs. item extremity. Educational and Psychological Measurement, 83(6), 1085–1112. https://doi.org/10.1177/00131644221143972
40. Karandashev, V. (2021). Cultural models of emotions. Springer.
41. Kline, T.J.B. (2005). Psychological testing: A practical approach to design and evaluation. Sage.
42. Knight, R.G., Chisholm, B.J., Marsh, N.V., & Godfrey, H.P.D. (1998). Some normative, reliability, and factor analytic data for the revised UCLA Loneliness Scale. Journal of Clinical Psychology, 44(2), 203 206. https://doi.org/10.1002/1097 4679(198803)44:2<203::AID-JCLP2270440218>3.0.CO;2-5
43. Kula Kartal, S. (2021). Examining scale items in terms of method effects based on the bifactor item response theory model. Kastamonu Education Journal, 29(1), 201–209. https://doi.org/10.24106/kefdergi.708968
44. Kula Kartal, S., Aybek, E.C., & Yaşar, M. (2022). Investigating the wording effect in scales based on the different dimension reduction techniques. Journal of Uludag University Faculty of Education, 35(1), 44–67. https://doi.org/10.19171/uefad.1033284
45. Locker, D., Jokovic, A., & Allison, P. (2013). Direction of wording and responses to items in oral health related quality of life questionnaires for children and their parents. Community Dentistry Oral Epidemiology, 35(4), 255–262. http://dx.doi.org/10.1111/j.1600-0528.2007.00320.x
46. Marsh, H.W. (1986). Negative item bias in ratings scales for preadolescent children: A cognitive developmental phenomenon. Developmental Psychology, 22(1), 37 49. https://doi.org/10.1037/0012-1649.22.1.37
47. Marsh, H.W. (1996). Positive and negative global self- esteem: A substantively meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70(4), 810–819. https://doi.org/10.1037/0022-3514.70.4.810
48. Marsh, H.W., & Hocevar, D. (1985). Application of confirmatory factor analysis to the study of self-concept: First- and higher order factor models and their invariance across groups. Psychological Bulletin, 97(3), 562–582. https://doi.org/10.1037/0033-2909.97.3.562
49. Mayerl, J., & Giehl, C. (2018). A closer look at attitude scales with positive and negative items response latency perspectives on measurement quality. Survey Research Methods, 12(3), 193–209. https://doi.org/10.18148/srm/2018.v12i3.7207
50. Meyers, L.S, Gamst, G., & Guarino, A.J. (2006). Applied multivariate research: Design and interpretation. Sage.
51. Michaelides, M.P. (2019). Negative keying effects in the factor structure of TIMSS 2011 motivation scales and associations with reading achievement. Applied Measurement in Education, 32(4), 365–378. https://doi.org/10.1080/08957347.2019.1660349
52. Pallant, J. (2005). SPSS survival manual (2nd ed.). Allen & Unwin.
53. Pilotte, W.J., & Gable R.K. (1990). The impact of positive and negative item stems on the validity of a computer anxiety scale. Educational and Psychological Measurement, 50(3), 603–610. http://dx.doi.org/10.1177/0013164490503016
54. Podsakoff, P.M., MacKenzie, S.B., Lee J.-Y., Podsakoff, N.P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879 903. https://doi.org/10.1037/0021 9010.88.5.879
55. Roszkowski, M.J., & Soven, M. (2010). Shifting gears: Consequences of including two negatively worded items in the middle of a positively worded questionnaire. Assessment & Evaluation in Higher Education, 35(1), 113 130. http://dx.doi.org/10.1080/02602930802618344
56. Salazar, M.S. (2015). The dilemma of combining positive and negative items in scales. Psicothema, 27(2), 192–200. https://doi.org/10.7334/psicothema2014.266
57. Pituch, K.A., & Stevens, J.P. (2016). Applied multivariate statistics for the social sciences (6th ed.). Routledge.
58. Schmitt, D.P., & Allik, J. (2005). Simultaneous administration of the Rosenberg self-esteem scale in 53 nations: Exploring the universal and culture-specific features of global self-esteem. Journal of Personality and Social Psychology, 89(4), 623 642. https://doi.org/10.1037/0022-3514.89.4.623
59. Schotte, C.K.W., Maes, M., Cluydts, R., & Cosyns, P. (1996). Effects of affective-semantic mode of item presentation in balanced self-report scales: Biased construct validity of the Zung self-rating depression scale. Psychological Medicine, 26(6), 1161–1168. http://dx.doi.org/10.1017/s0033291700035881
60. Schriesheim, C.A., Eisenbach, R.J., & Hill, K.D. (1991). The effect of negation and polar opposite item reversals on questionnaire reliability and validity: An experimental investigation. Educational and Psychological Measurement, 51(1), 67 78. https://doi.org/10.1177/0013164491511005
61. Schriesheim, C.A., & Hill, K.D. (1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. Educational and Psychological Measurement, 41(4), 1101–1114. http://dx.doi.org/10.1177/001316448104100420
62. Spector, P.E., Van Katwyk, P.T., Brannick, M.T., & Chen, P.Y. (1997). When two factors don’t reflect two constructs: How item characteristics can produce artifactual factors. Journal of Management, 23(5), 659–677. https://doi.org/10.1016/S0149-2063(97)90020-9
63. Stauesberg, M. (2002). Survey and questionnaires. In S. Engler & Micheal Stausber (Eds,), The Routledge handbook of research methods in the study of religion (pp. 461–482). Routledge.
64. Stewart, T.J., & Frye, A.W. (2004). Investigating the use of negatively phrased survey items in medical education settings: Common wisdom or common mistake? Academic Medicine, 79(10), 18–20. https://doi.org/10.1097/00001888-200410001-00006
65. Suárez-Alvarez, J., Pedrosa, I., Lozano, L.M., García-Cueto, E., Cuesta, M., & Muñiz, J. (2018). Using reversed items in Likert scales: A questionable practice. Psicothema, 30(2), 149–158. https://doi.org/10.7334/psicothema2018.33
66. Swain, S.D., Weather, D., & Niedrich, R.W. (2008). Assessing three sources of misresponse to reversed Likert items. Journal of Marketing Research, 45(1), 116 131. https://doi.org/10.1509/jmkr.45.1.116
67. van Sonderen, E., Sanderman, R., & Coyne, J.C. (2013). Ineffectiveness of reverse wording of questionnaire items: Let’s learn from cows in the rain. PLoS ONE, 8(7), 1 7. https://doi.org/10.1371/journal.pone.0068967
68. Verešová, M., & Malá, D. (2016, 11–15October). Attitude toward school and learning and academic achievement of adolescents [Conference Presentation]. 7th International Conference on Education and Educational Psychology. https://www.europeanproceedings.com/article/10.15405/epsbs.2016.11.90
69. Vigil-Colet, A., Navarro-González, D., & Morales-Vives, F. (2020). To reverse or to not reverse Likert type items: That is the question. Psicothema, 32(1), 108 114. https://doi.org/10.7334/psicothema2019.286
70. Weems, G.H., Onwuegbuzie, A.J., & Lustig, D. (2003) Profiles of respondents who respond inconsistently to positively- and negatively-worded items on rating scales. Evaluation & Research in Education, 17(1), 45–60. http://dx.doi.org/10.1080/14664200308668290
71. Weems, G.H., Onwuegbuzie, A.J., & Collins, K.M.T. (2006) The role of reading comprehension in responses to positively and negatively worded items on rating scales. Evaluation & Research in Education, 19(1), 3 20. https://doi.org/10.1080/09500790608668322
72. Weijters, B., & Baumgartner, H. (2012). Misresponse to reversed and negated items in surveys: A review. Journal of Marketing Research, 49(5), 737 747. https://doi.org/10.1509/jmr.11.0368
73. Wong, N., Rindfleisch, A., & Burroghs, J.E. (2003). Do reversed-worded items confound measures in cross-cultural consumer research? The case of material values scale. Journal of Consumer Research, 30(1), 72–91. https://doi.org/10.1086/374697
74. Zhang, X., Noor R., & Savalei, V. (2016) Examining the effect of reverse worded items on the factor structure of the need for cognition scale. PLoS ONE, 11(6), 1 15. https://doi.org/10.1371/journal.pone.0157795