The Effect of Option Differences on Psychometric Properties of Items in Likert-Type Scales

Year 2023, , 207 - 237, 10.09.2023

Nuri Doğan , Meltem Yurtçu , Ceylan Gündeğer

https://doi.org/10.19126/suje.1253876

Abstract

Likert-type scales are often used in education and psychology. In Likert-type scales, response options/categories, like items, are expected not to direct individuals’ responses. Although the researchers themselves make decision on how to arrange categories during scale development, it is possible that different categories reveal different response behaviors. In the literature, it has been observed that differentiations in the number of categories of forms are studied more, yet there are a limited number of studies investigating the middle category in the forms with different labels. Furthermore, it has also been observed that there are limited number of empirical studies conducted based on polytomous Item Response Theory. This study, which was conducted to close this gap in the literature, was carried out with 377 students. The options of the attitude scale were denominated with different labels, and thus four different forms were generated. Only the middle category names were changed in the first three forms, and in the fourth form, the categories were graded. The data obtained from the forms were analyzed using the Graded Response Model and the Generalized Partial Credit Model depending on Item Response Theory. After the examination of reliability of the forms, the parameters in these forms, and the relationships between the parameters according to both models, inferences were made as to how the differences of the middle category in the forms had an effect on the perceptions of individuals.

Keywords

Likert type scales, scale categories, perception, item response theory models

Supporting Institution

-

Project Number

-

Thanks

-

LİKERT TİPİ ÖLÇEKLERDE SEÇENEK FARKLILIKLARININ MADDELERİN PSİKOMETRİK ÖZELLİKLERİNE ETKİSİ

Abstract

Eğitimde ve psikolojide sıklıkla Likert tipi ölçeklerden yararlanılmaktadır. Likert tipi ölçeklerde maddeler gibi cevap seçeneklerinin/kategorilerin de bireyin cevabını yönlendirmemesi beklenir. Ölçek geliştirme aşamasında kategorilerin nasıl düzenleneceğine araştırmacı kendisi karar verse de farklı kategorilerin farklı cevaplama davranışı ortaya çıkarması mümkündür. Alanyazında formlara ait kategori sayısındaki farklılaşmaların daha sık incelendiği ve formlardaki orta kategorinin farklı etiketlerle ele alınmasına ait sınırlı sayıda çalışmanın yer aldığı gözlemlenmiştir. Ayrıca çok kategorili madde tepki kuramı temelinde yürütülen ampirik araştırmaların az sayıda olduğu görülmektedir. Literatürdeki bu açığı kapatmaya yönelik olarak gerçekleştirilen bu çalışma 377 öğrenci üzerinde gerçekleştirilmiştir. Uygulamada tutum ölçeğine ait sorular farklı etiketlerle isimlendirilerek dört farklı form oluşturulmuştur. İlk üç formda sadece orta kategori isimleri farklılaştırılmış, dördüncü formun ise kategorileri dereceleme türünden verilmiştir. Formlardan elde edilen veri, Madde Tepki Kuramı temelinde Graded Response Model ve Generalized Partial Credit Model ile değerlendirilmiştir. Her iki modele göre formlara ait güvenirlikler, formlardaki parametreler ve parametreler arasındaki ilişkiler incelendikten sonra orta değerin formlardaki farklılıklarının bireylerin algıları üzerinde nasıl bir etki oluşturduğuna yönelik olarak çıkarımlara ulaşılmıştır.

Keywords

Likert tipi ölçekler, ölçek kategorileri, algı, madde tepki kuramı modelleri

Project Number

-

References

• Albaum, G. (1997). The Likert scale revisited: An alternate version. Journal of the Market Research Society, 39(2), 331-342. https://doi.org/10.1177/147078539703900202
• Adelson, J. L., & McCoach, D. B. (2010). Measuring the mathematical attitudes of elementary students: The effects of a 4-point or 5-point Likert-type scale. Educational and Psychological Measurement, 70(5), 796–807. https://doi.org/10.1177/0013164410366694
• Andersson. B., & Xin, T. (2018). Large sample confidence intervals for item response theory reliability coefficients. Educational Psychological Measurement, 78(1), 32-45. https://doi.org/10.1177/0013164417713570
• Annett, J. (2002). Subjective rating scales: Science or art? Ergonomics, 45, 966-987. https://doi.org/10.1080/00140130210166951
• Blumberg, H. H., DeSoto, C. B. & Kuethe, J. L. (1966). Evaluation of rating scale formats. Personnel Psychology, 19, 243-259. https://doi.org/10.1111/j.1744-6570.1966.tb00301.x
• Büyükkıdık, S., & Atar, H. (2018). Çok kategorili item tepki kuramı modellerinin örneklem büyüklüğü açısından incelenmesi [Examining multi-category item response theory models in terms of sample size]. Gazi Üniversitesi Gazi Eğitim Fakültesi Dergisi, 38(2), 663-692. https://doi.org/10.17152/gefad.334608
• Bartolucci, F., Bacci, S., & Gnaldi, M. (2015). Statistical analysis of questionnaires: A unified approach based on R and Stata. Boca Raton, FL: Chapman and Hall/CRC.
• Carle, A. C., Jaffee, D., Vaughan, N. W., & Eder, D. (2009). Psychometric properties of three new national survey of student engagement based engagement scales: An item response theory analysis. Research in Higher Education, 50, 775-794. https://doi.org/10.1007/s11162-009-9141-z
• Chalmers, R. P. (2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29. https://doi.org/10.18637/jss.v048.i06
• Chyung, S. Y., Roberts, K., Swanson, I., & Hankinson, A. (2017). Evidence‐based survey design: The use of a midpoint on the Likert scale. Performance Improvement, 56(10), 15-23. https://doi.org/10.1002/pfi.21727
• Cordier, R., Munro, N., Wilkes-Gillan, S., Speyer, R., Parsons, L., & Joosten, A. (2019). Applying Item Response Theory (IRT) modeling to an observational measure of childhood pragmatics: The pragmatics observational measure-2. Frontiers in Psychology, 10, 408. https://doi.org/10.3389/fpsyg.2019.00408
• Croasmun, J. T., & Ostrom, L. (2011). Using Likert-type scales in the social sciences. Journal of adult education, 40(1), 19-22. Retrieved from https://eric.ed.gov/?id=EJ961998
• Dai, S., Vo, T. T., Kehinde, O. J., He, H., Xue, Y., Demir, C., & Wang, X. (2021, September). Performance of polytomous IRT models with rating scale data: An investigation over sample size, instrument length, and missing data. In Frontiers in Education (Vol. 6, p. 721963). Frontiers Media SA. https://doi.org/10.3389/feduc.2021.721963
• Dixon, P. N., Bobo, M., & Stevick, R. A. (1984). Response differences and preferences for all category defined and end-defined Likert formats. Educational & Psychological Measurement, 44, 61-66. https://doi.org/10.1177/0013164484441006
• Dunkel, A. (2015). Visualizing the perceived environment using crowdsourced photo geodata. Landscape and urban planning, 142, 173-186. https://doi.org/10.1016/j.landurbplan.2015.02.022
• Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Psychology Press.
• Erkuş, A. (2012). A measurement and scale development in Psychology I: Basic concepts and processes. Ankara: Pegem Academy Publishing (In Turkish).
• Finch, H., & French, B. F. (2019). A comparison of estimation techniques for IRT models with small samples. Applied Measurement in Education, 32(2), 77–96. https://doi.org/10.1080/08957347.2019.1577243
• Finn, R. H. (1972). Effects of some variations of rating scale characteristics on the means and reliabilities of ratings. Educational & Psychological Measurement, 32, 255-265. https://doi.org/10.1177/001316447203200203
• Gibson, J.L., Ivancevich, J.M., James H., & Donnely Jr. (1996), Organizational behavior structure, Process. 9th Edition, Chicago: Irwin.
• Hambleton, R. K., & Swaminathan, H. (1985). Item Response Theory: Principles and Applications. Boston: Kluwer Nijhoff Publishing.
• Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory and health outcomes measurement in the 21st century. Medical care, 38(9), 28-42. https://doi.org/10.1097%2F00005650-200009002-00007
• Huang, H. Y. (2016). Mixture random-effect IRT models for controlling extreme response style on rating scales. Frontiers in Psychology, 7(1706), 1-15. https://doi.org/10.3389/fpsyg.2016.01706
• Hulın, C. L., Drasgow, F., & Parsons, C. K. (1983). Item response theory: Application to psychological measurement. Homewood, IL: Dow Jones-Irwin.
• Jacko, E. J., & Huck, S. W. (1974, April). The Effect of varying the response format on the statistical characteristics of the Alpert-Haber Achievement Anxiety Test. Paper presented at the Annual Meeting of the American Educational Research Association (59th, Chicago, Illinois).
• Jin, K. Y., & Wang, W. C. (2014). Item response theory models for performance decline during testing. Journal of Educational Measurement, 51, 178–200. https://doi.org/10.1111/jedm.12041
• Kieftenbeld, V., & Natesan, P. (2012). Recovery of graded response model parameters. Applied Psychological Measurement, 36(5), 399–419. https://doi.org/10.1177/0146621612446170
• Korkmaz, S., Goksuluk, D., & Zararsiz, G. (2014). MVN: An R package for assessing multivariate normality. The R Journal, 6(2),151-162. https://doi.org/10.32614/RJ-2014-031
• Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajewski, B. J., Hróbjartsson, A., ... & Streiner, D. L. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. International journal of nursing studies, 48(6), 661-671. https://doi.org/10.1016/j.ijnurstu.2011.01.016
• Krosnick, J. A., & Berent, M. K. (1993). Comparisons of party identification and policy preferences: The impact of survey question format. American Journal of Political Science, 37, 941–964. https://doi.org/10.2307/2111580 Kulas, J. T., & Stachowski, A. A. (2009). Middle category endorsement in odd-numbered Likert response scales: Associated item characteristics, cognitive demands, and preferred meanings. Journal of Research in Personality, 43(3), 489-493. https://doi.org/10.1016/j.jrp.2008.12.005
• Lange, T., Schmitt, J., Kopkow, C., Rataj, E., Günther, K. P., & Lützner, J. (2017). What do patients expect from total knee arthroplasty? A Delphi consensus study on patient treatment goals. The Journal of arthroplasty, 32(7), 2093-2099. https://doi.org/10.1016/j.arth.2017.01.053
• Mendiburu, F. D. (2021). Agricolae: Statistical Procedures for Agricultural Research. 2017. R package version, 1-1.
• Moors, G. (2008). Exploring the effect of a middle response category on response style in attitude measurement. Quality & quantity, 42, 779-794. https://doi.org/10.1007/s11135-006-9067-x
• Muraki, E. (1992). A generalized partial credit model: Application of an em algorithm. ETS research report-1, i-30. https://doi.org/10.1002/j.2333-8504.1992.tb01436.x
• Nartgün, Z. (2002). Aynı tutumu ölçmeye yönelik Likert tipi ölçek ile metrik ölçeğin Item ve ölçek özelliklerinin klasik test kuramı ve örtük özellikler kuramına göre incelenmesi. [Examining the Item and scale properties of Likert-type scale and metric scale for measuring the same attitude according to classical test theory and latent trait theory.] Hacettepe Üniversitesi Sosyal Bilimler Enstitüsü, Yayımlanmamış Doktora Tezi, Ankara.
• Newstead, S. E. & Arnold, J. (1989). The effect of response format on ratings of teaching. Educational & Psychological Measurement, 49, 33-43. https://doi.org/10.1177/0013164489491004
• OECD (2021). PISA 2018 Technical Report. Paris: Organization for Economic Cooperation and Development (OECD). https://www.oecd.org/pisa/data/pisa2018technicalreport/ Ostini, R. & Nering, M. L. (2006). Polytomous item response theory models. California: Sage.
• Ogle, D. H., Wheeler, P. & Dinno, A. (2021). FSA: Fisheries Stock Analysis. R package version 0.9.0, Retrieved from https://github.com/droglenc/FSA.
• O’Muircheartaigh, C., Krosnick, J. A., & Helic, A. (2000). Middle alternatives, acquiescence, and the quality of questionnaire data. The Center for Advanced Study in the Behavioral Sciences. Retrieved from:https://www.academia.edu/18408388/Middle_Alternatives_Acquiescence_and_the_Quality_Questionnaire_Data?bulkDownload=thisPaper-topRelated-sameAuthor-citingThis-citedByThis-secondOrderCitations&from=cover_page
• Pomerantz, J. R. (2003). Perception: Overview. In: Lynn Nadel (Ed.), Encyclopedia of Cognitive Science, Vol. 3, London: Nature Publishing Group, pp. 527–537.
• R Development Core Team. (2013). R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.Rproject.org
• Rajamanickam M (2007) Modern general psychology thoroughly revised and expanded, 2nd edn. Concept Publishing Company, New Delhi, p.330
• Robitzsch, A., Kiefer, T., & Wu, M. (2021). TAM: Test Analysis Modules. R package version 3.7 16, Retrieved from: https://CRAN.R-project.org/package=TAM
• Qiong, O. U. (2017). A brief introduction to perception. Studies in Literature and Language, 15(4), 18 -28. https://doi.org/10.3968/10055
• Samejima, F. (1969). Estimation of latent trait ability using a response pattern of graded scores. Psychometrika Monograph Supplement, No.17.
• Schneider, L., Chalmers, R. P., Debelak, R. &. Merkle, E. C. (2020) Model selection of nested and non nested item response models using vuong tests, Multivariate Behavioral Research, 55(5), 664-684, https://doi.org/10.1080/00273171.2019.1664280
• Sischka, P. E., Costa, A. P., Steffgen, G., & Schmidt, A. F. (2020). The WHO-5 well-being index validation based on item response theory and the analysis of measurement invariance across 35 countries. Journal of Affective Disorders Reports, 1, 100020. https://doi.org/10.1016/j.jadr.2020.100020
• Sözer, E., & Kahraman, N. (2021). Investigation of psychometric properties of Likert items with the same response categories using polytomous item response theory models. Journal of Measurement and Evaluation in Education and Psychology, 12(2), 129-146. https://doi.org/10.21031/epod.819927
• Steiner, M., & Grieder, S. (2020). EFAtools: An R package with fast and flexible implementations of exploratory factor analysis tools. Journal of Open Source Software, 5(53), 2521. https://doi.org/10.21105/joss.02521
• Sung, H. J., & Kang, T. (2006, April). Choosing a polytomous IRT model using Bayesian model selection methods. In National Council on Measurement in Education Annual Meeting (pp. 1-36).
• Tabachnick, B. G., & Fidell, L. S. (2007). Using Multivariate Statistics. USA: Pearson Education Inc.
• Yaşar, M., & Aybek, E. C. (2019). Üniversite öğrencileri için bir yılmazlık ölçeğinin geliştirilmesi: Item tepki kuramı temelinde geçerlilik ve güvenilirlik çalışması [Development of a resilience scale for university students: A validity and reliability study based on item response theory]. İlköğretim Online, 18(4), 1687 -1699. Retrieved from https://ilkogretim-online.org/fulltext/218-1597121020.pdf?1618815938
• Wakita, T., Ueshima, N., & Noguchi, H. (2012). Psychological distance between categories in the Likert scale: Comparing different numbers of options. Educational and Psychological Measurement, 72, 533-546. https://doi.org/10.1177/0013164411431162
• Wetzel, E., Böhnke, J. R., & Rose, N. (2016). A simulation study on methods of correcting for the effects of extreme response style. Educational and Psychological Measurement, 76(2), 304-324. https://doi.org/10.1177/0013164415591848
• Wyatt, R. C., & Meyers, L. S. (1987). Psychometric properties of four-point Likert-type response scales. Educational & Psychological Measurement, 47, 27-35. https://doi.org/10.1177/0013164487471003