The effects of reverse items on psychometric properties and respondents’ scale scores according to different item reversal strategies

Year 2024, Volume: 11 Issue: 1, 20 - 38, 16.03.2024

Mustafa İlhan Neşe Güler Gülşen Taşdelen Teker Ömer Ergenekon

https://doi.org/10.21449/ijate.1345549

Abstract

This study aimed to examine the effects of reverse items created with different strategies on psychometric properties and respondents’ scale scores. To this end, three versions of a 10-item scale in the research were developed: 10 positive items were integrated in the first form (Form-P) and five positive and five reverse items in the other two forms. The reverse items in the second and third forms were crafted using antonyms (Form-RA) and negations (Form-RN), respectively. Based on the research results, Form-P was unidimensional, while other forms were two-dimensional. Moreover, although reliability coefficients of all forms were obtained as above .80, the lowest one was acquired for Form-RN. There were strong-positive relationships between students’ scores in the three scale forms. However, the lowest one was estimated between Form-P and Form-RN. Finally, there was a significant difference between the students’ mean scores obtained from Form–RN and other two versions, but the effect size of the said difference was small. In conclusion, all these results indicate that different types of reverse items influence psychometric properties and respondents’ scale scores differently.

Keywords

Antonyms, Negations, Item reversal strategy, Item wording, Reverse worded items

Ethical Statement

Dicle University, 27.12.2021-21445.

References

• Ahlawat, K.S. (1985). On the negative valence items in self-report measures. The Journal of General Psychology, 112(1), 89–99. https://doi.org/10.1080/00221309.1985.9710992
• Aybek, E.C. (2021). Data preparation for factor analysis. URL: https://shiny.eptlab.com/dp2fa/
• Bandalos, D.L. (2018). Measurement theory and applications for the social sciences. The Guilford.
• Benson, J., & Hocevar, D. (1985). The impact of item phrasing on the validity of attitude scales for elementary school children. Journal of Educational Measurement, 22(3), 231–240. https://doi.org/10.1111/j.1745-3984.1985.tb01061.x
• Bergstrom, B.A., & Lunz, M.E. (1998, 13–17, April). Rating scale analysis: Gauging the impact of positively and negatively worded items. Annual Meeting of the American Educational Research Association, San Diego.
• Boley, B.B., Jordan, E., & Woosnam, K.M. (2021). Reversed polarity items in tourism scales: Best practice or dimensional pitfall? Current Issues in Tourism, 24(4), 1 13. https://doi.org/10.1080/13683500.2020.1774517
• Bolt, D., Wang, Y.C., Meyer, R.H., & Pier, L. (2020). An IRT mixture model for rating scale confusion associated with negatively worded items in measures of social-emotional learning. Applied Measurement in Education, 33(4), 1 18. https://doi.org/10.1080/08957347.2020.1789140
• Brace, I., & Bolton, I. (2022). Questionnaire design: How to plan, structure and write survey material for effective market research (5th ed.). Kogan Page.
• Bulut, H.Ç. (2021). Item wording effects in psychological measures: Do early literacy skills matter? Journal of Measurement and Evaluation in Education and Psychology, 12(3), 239-253. https://doi.org/10.21031/epod.944067
• Bulut, H.Ç., & Bulut, O. (2022). Item wording effects in self-report measures and reading achievement: Does removing careless respondents help? Studies in Educational Evaluation, 72(2). https://doi.org/10.1016/j.stueduc.2022.101126
• Carlson, M., Wilcox, R., Chou, C.-P., Chang, M., Yang, F., Blanchard, J., Marterella, A., Kuo, A., & Clark, F. (2011). Psychometric properties of reverse-scored items on the CES-D in a sample of ethnically diverse older adults. Psychological Assessment, 23(2), 558–562. https://doi.org/10.1037/a0022484
• Chamberlain, V.M., & Cummings, M.N. (1984). Development of an instructor/course evaluation instrument. College Student Journal, 18(3), 246–250.
• Chyung, S.Y.Y., Barkin, J.R., & Shamsy, J.A. (2018). Evidence-based survey design: The use of negatively worded items in surveys. Performance Improvement, 57(3), 16–25. https://doi.org/10.1002/pfi.21749
• Coleman, C.M. (2013). Effects of negative keying and wording in attitude measures: A mixed-methods study [Unpublished doctoral dissertation, James Madison University]. https://commons.lib.jmu.edu/diss201019/73/
• Conrad, K.J., Wright, B.D., McKnight P., McFall, M., Fontana, A., & Rosenheck, R. (2004). Comparing traditional and Rasch analyses of the Mississippi PTSD Scale: Revealing limitations of reverse-scored items. Journal of Applied Measurement, 5(1), 15–30.
• Coolican, H. (2013). Research methods and statistics in psychology (5th ed.). Routledge.
• Dagnew, A. (2017). The relationship between students’ attitudes towards school, values of education, achievement motivation and academic achievement in Gondar secondary schools, Ethiopia. Research in Pedagogy, 7(1), 30–42. https://doi.org/10.17810/2015.46
• Dawson, J. (2017). Analysing quantitative survey data for business and management students. Sage.
• DeVellis, R. (2017). Scale development: Theory and applications (4th ed.). Sage.
• Diedenhofen, B., & Musch, J. (2015). Cocor: A comprehensive solution for the statistical comparison of correlations. PLoSONE, 10(4), 1 12. https://doi.org/10.1371/journal.pone.0121945
• Dooden, H. (2014). The effects of positively and negatively worded items on the factor structure of the UCLA loneliness scale. Journal of Psychoeducational Assessment, 33(3), 259–267. https://doi.org/10.1177/0734282914548325
• Dueber, D.M., Toland, M.D., Lingat, J.E., Love, A.M.A., Qiu, C., Wu, R., & Brown, A.V. (2021). To Reverse item orientation or not to reverse item orientation, that is the question. Assessment, 29(7), 1422–1440. https://doi.org/10.1177/10731911211017635
• Dunbar, M., Ford, G., Hunt, K., & Der, G. (2000). Question wording effects in the assessment of global self-esteem. European Journal of Psychological Assessment, 16(1), 13–19. https://doi.org/10.1027//1015-5759.16.1.13
• Ebesutani, C., Drescher, C.F., Reise, S.P., Heiden, L., High, T.L., Damon, J.D., & Young, J. (2012). The loneliness questionnaire-short version: An evaluation of reverse- worded and non-reverse-worded items via item response theory. Journal of Personality Assessment, 94(4), 427–437. https://doi.org/10.1080/00223891.2012.662188
• Fabrigar, L.R., Wegener, D.T., MacCallum, R.C., & Strahan, E.J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299. http://dx.doi.org/10.1037/1082-989X.4.3.272
• Finney, S.J. (2001). A comparison of the psychometric properties of negatively and positively worded questionnaire items (Publication No. 3009716) [Doctoral dissertation, University of Nebraska]. ProQuest Dissertations & Theses Global.
• Furr, R.M. (2018). Psychometrics: An introduction (3rd ed.). Sage.
• Geddes, J.D., Murrell, A.R., & Bauguss, J. (2010). Childhood learning: An examination of ability and attitudes toward school. Creative Education, 1(3), 170 183. https://doi.org/10.4236/ce.2010.13027
• Gnambs, T., & Schroeders, U. (2020). Cognitive abilities explain wording effects in the Rosenberg Self Esteem Scale. Assessment, 27(2), 404 418. https://doi.org/10.1177/1073191117746503
• Greenberger, E., Chen, C., Dmitrieva, J., & Farruggia, S.P. (2003). Item-wording and the dimensionality of the Rosenberg Self-Esteem Scale: Do they matter? Personality and Individual Differences, 35(6), 1241 1254. https://doi.org/10.1016/S0191 8869(02)00331-8
• Guyatt, G.H., Cook, D.J., King, D., Norman, G.R., Kane, S.L., & Van Ineveld, C. (1999). Effect of the framing of questionnaire items regarding satisfaction with training on residents’ responses. Academic Medicine, 74(2), 192 194. https://doi.org/10.1097/00001888 199902000-00018
• Hazlett-Stevens, H., Ullman, J.B., & Craske, M.G. (2004). Factor structure of the Penn State worry questionnaire: Examination of a method factor. Assessment, 11(4), 361–370. https://doi.org/10.1177/1073191104269872
• Herche, J., & England, B. (1996). Reversed-polarity items and scale unidimensionality. Journal of the Academy of Marketing Science, 24(4), 366 374. https://doi.org/10.1177/0092070396244007
• Hooper, M., Arora, A., Martin, M.O., & Mullis, I.V.S, (2013, 24–25 June). Examining the behavior of “reverse directional” items in the TIMSS 2011 context questionnaire scales [Conference presentation]. 5th IEA International Research Conference. National Institute of Education, Nanyang Technological University, Singapore. https://www.iea.nl/sites/default/files/2019-04/IRC-2013_Hooper_etal.pdf
• Hughes, G.D. (2009). The impact of incorrect responses to reverse-coded survey items. Research in the Schools, 16(2), 76–88.
• Ilhan, M., & Guler, N. (2017). The number of response categories and the reverse scored item problem in Likert-type scales: A study with the Rasch model. Journal of Measurement and Evaluation in Education and Psychology, 8(3), 321 343. http://dx.doi.org/10.21031/epod.321057
• JASP Team (2022). JASP (Version 0.16.3) [Computer software].
• Johnson, J.M., Bristow, D.N., & Schneider, K.C. (2004). Did you not understand the question or not? An investigation of negatively worded questions in survey research. Journal of Applied Business Research, 20(1), 75–86. https://doi.org/10.19030/jabr.v20i1.2197
• Kam, C.C.S. (2023). Why do regular and reversed items load on separate factors? Response difficulty vs. item extremity. Educational and Psychological Measurement, 83(6), 1085–1112. https://doi.org/10.1177/00131644221143972
• Karandashev, V. (2021). Cultural models of emotions. Springer.
• Kline, T.J.B. (2005). Psychological testing: A practical approach to design and evaluation. Sage.
• Knight, R.G., Chisholm, B.J., Marsh, N.V., & Godfrey, H.P.D. (1998). Some normative, reliability, and factor analytic data for the revised UCLA Loneliness Scale. Journal of Clinical Psychology, 44(2), 203 206. https://doi.org/10.1002/1097 4679(198803)44:2<203::AID-JCLP2270440218>3.0.CO;2-5
• Kula Kartal, S. (2021). Examining scale items in terms of method effects based on the bifactor item response theory model. Kastamonu Education Journal, 29(1), 201–209. https://doi.org/10.24106/kefdergi.708968
• Kula Kartal, S., Aybek, E.C., & Yaşar, M. (2022). Investigating the wording effect in scales based on the different dimension reduction techniques. Journal of Uludag University Faculty of Education, 35(1), 44–67. https://doi.org/10.19171/uefad.1033284
• Locker, D., Jokovic, A., & Allison, P. (2013). Direction of wording and responses to items in oral health related quality of life questionnaires for children and their parents. Community Dentistry Oral Epidemiology, 35(4), 255–262. http://dx.doi.org/10.1111/j.1600-0528.2007.00320.x
• Marsh, H.W. (1986). Negative item bias in ratings scales for preadolescent children: A cognitive developmental phenomenon. Developmental Psychology, 22(1), 37 49. https://doi.org/10.1037/0012-1649.22.1.37
• Marsh, H.W. (1996). Positive and negative global self- esteem: A substantively meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70(4), 810–819. https://doi.org/10.1037/0022-3514.70.4.810
• Marsh, H.W., & Hocevar, D. (1985). Application of confirmatory factor analysis to the study of self-concept: First- and higher order factor models and their invariance across groups. Psychological Bulletin, 97(3), 562–582. https://doi.org/10.1037/0033-2909.97.3.562
• Mayerl, J., & Giehl, C. (2018). A closer look at attitude scales with positive and negative items response latency perspectives on measurement quality. Survey Research Methods, 12(3), 193–209. https://doi.org/10.18148/srm/2018.v12i3.7207
• Meyers, L.S, Gamst, G., & Guarino, A.J. (2006). Applied multivariate research: Design and interpretation. Sage.
• Michaelides, M.P. (2019). Negative keying effects in the factor structure of TIMSS 2011 motivation scales and associations with reading achievement. Applied Measurement in Education, 32(4), 365–378. https://doi.org/10.1080/08957347.2019.1660349
• Pallant, J. (2005). SPSS survival manual (2nd ed.). Allen & Unwin.
• Pilotte, W.J., & Gable R.K. (1990). The impact of positive and negative item stems on the validity of a computer anxiety scale. Educational and Psychological Measurement, 50(3), 603–610. http://dx.doi.org/10.1177/0013164490503016
• Podsakoff, P.M., MacKenzie, S.B., Lee J.-Y., Podsakoff, N.P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879 903. https://doi.org/10.1037/0021 9010.88.5.879
• Roszkowski, M.J., & Soven, M. (2010). Shifting gears: Consequences of including two negatively worded items in the middle of a positively worded questionnaire. Assessment & Evaluation in Higher Education, 35(1), 113 130. http://dx.doi.org/10.1080/02602930802618344
• Salazar, M.S. (2015). The dilemma of combining positive and negative items in scales. Psicothema, 27(2), 192–200. https://doi.org/10.7334/psicothema2014.266
• Pituch, K.A., & Stevens, J.P. (2016). Applied multivariate statistics for the social sciences (6th ed.). Routledge.
• Schmitt, D.P., & Allik, J. (2005). Simultaneous administration of the Rosenberg self-esteem scale in 53 nations: Exploring the universal and culture-specific features of global self-esteem. Journal of Personality and Social Psychology, 89(4), 623 642. https://doi.org/10.1037/0022-3514.89.4.623
• Schotte, C.K.W., Maes, M., Cluydts, R., & Cosyns, P. (1996). Effects of affective-semantic mode of item presentation in balanced self-report scales: Biased construct validity of the Zung self-rating depression scale. Psychological Medicine, 26(6), 1161–1168. http://dx.doi.org/10.1017/s0033291700035881
• Schriesheim, C.A., Eisenbach, R.J., & Hill, K.D. (1991). The effect of negation and polar opposite item reversals on questionnaire reliability and validity: An experimental investigation. Educational and Psychological Measurement, 51(1), 67 78. https://doi.org/10.1177/0013164491511005
• Schriesheim, C.A., & Hill, K.D. (1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. Educational and Psychological Measurement, 41(4), 1101–1114. http://dx.doi.org/10.1177/001316448104100420
• Spector, P.E., Van Katwyk, P.T., Brannick, M.T., & Chen, P.Y. (1997). When two factors don’t reflect two constructs: How item characteristics can produce artifactual factors. Journal of Management, 23(5), 659–677. https://doi.org/10.1016/S0149-2063(97)90020-9
• Stauesberg, M. (2002). Survey and questionnaires. In S. Engler & Micheal Stausber (Eds,), The Routledge handbook of research methods in the study of religion (pp. 461–482). Routledge.
• Stewart, T.J., & Frye, A.W. (2004). Investigating the use of negatively phrased survey items in medical education settings: Common wisdom or common mistake? Academic Medicine, 79(10), 18–20. https://doi.org/10.1097/00001888-200410001-00006
• Suárez-Alvarez, J., Pedrosa, I., Lozano, L.M., García-Cueto, E., Cuesta, M., & Muñiz, J. (2018). Using reversed items in Likert scales: A questionable practice. Psicothema, 30(2), 149–158. https://doi.org/10.7334/psicothema2018.33
• Swain, S.D., Weather, D., & Niedrich, R.W. (2008). Assessing three sources of misresponse to reversed Likert items. Journal of Marketing Research, 45(1), 116 131. https://doi.org/10.1509/jmkr.45.1.116
• van Sonderen, E., Sanderman, R., & Coyne, J.C. (2013). Ineffectiveness of reverse wording of questionnaire items: Let’s learn from cows in the rain. PLoS ONE, 8(7), 1 7. https://doi.org/10.1371/journal.pone.0068967
• Verešová, M., & Malá, D. (2016, 11–15October). Attitude toward school and learning and academic achievement of adolescents [Conference Presentation]. 7th International Conference on Education and Educational Psychology. https://www.europeanproceedings.com/article/10.15405/epsbs.2016.11.90
• Vigil-Colet, A., Navarro-González, D., & Morales-Vives, F. (2020). To reverse or to not reverse Likert type items: That is the question. Psicothema, 32(1), 108 114. https://doi.org/10.7334/psicothema2019.286
• Weems, G.H., Onwuegbuzie, A.J., & Lustig, D. (2003) Profiles of respondents who respond inconsistently to positively- and negatively-worded items on rating scales. Evaluation & Research in Education, 17(1), 45–60. http://dx.doi.org/10.1080/14664200308668290
• Weems, G.H., Onwuegbuzie, A.J., & Collins, K.M.T. (2006) The role of reading comprehension in responses to positively and negatively worded items on rating scales. Evaluation & Research in Education, 19(1), 3 20. https://doi.org/10.1080/09500790608668322
• Weijters, B., & Baumgartner, H. (2012). Misresponse to reversed and negated items in surveys: A review. Journal of Marketing Research, 49(5), 737 747. https://doi.org/10.1509/jmr.11.0368
• Wong, N., Rindfleisch, A., & Burroghs, J.E. (2003). Do reversed-worded items confound measures in cross-cultural consumer research? The case of material values scale. Journal of Consumer Research, 30(1), 72–91. https://doi.org/10.1086/374697
• Zhang, X., Noor R., & Savalei, V. (2016) Examining the effect of reverse worded items on the factor structure of the need for cognition scale. PLoS ONE, 11(6), 1 15. https://doi.org/10.1371/journal.pone.0157795