Exploring number of response categories in factor analysis: Implications for sample size

Abstract

Factor analysis is a statistical method to explore the relationships among observed variables and identify latent structures. It is crucial in scale development and validity analysis. Key factors affecting the accuracy of factor analysis results include the type of data, sample size, and the number of response categories. While some studies suggest that reliability improves with more response categories, others find no significant relationship between the number of response categories and reliability. A key consideration is that increasing the number of response categories can introduce measurement errors, especially when there are too many categories for participants to respond accurately. The study examines how different numbers of response categories affect sample size requirements in factor analysis, particularly under misspecified and correctly specified models. MonteCarloSEM package in R was used to simulate data sets based on sample size, number of response categories, model specification, and test length. Results show that a higher number of categories helps reduce bias and improve model fit, especially in smaller samples. However, when sample sizes are small or when fewer categories are used, increasing the number of items or the number of categories can improve parameter estimation. The findings suggest that for optimal results, researchers should carefully balance sample size, number of items, and response categories, particularly in studies with categorical data.

Keywords

• Factor analysis
• Sample size
• Number of response categories
• Number of items

References

1. Abulela, M.A.A., & Khalaf, M.A. (2024). Does the number of response categories impact validity evidence in self report measures? A scoping review. Sage Open, 14(1), 1 16. https://doi.org/10.1177/21582440241230363
2. Abdelsamea, M. (2020). The effect of the number of response categories on the assumptions and outputs of item exploratory and confirmatory factor analyses of measurement instruments in psychological research. Journal of Education Sohag UNV, 76, 1153-1222. https://doi.org/10.21608/edusohag.2020.103373
3. Bandalos, D.L., & Enders, C.K. (1996). The effects of nonnormality and number of response categories on reliability. Applied Measurement in Education, 9(2), 151 160. https://doi.org/10.1207/s15324818ame0902_4
4. Flora, D.B., & Curran, P.J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466-491. https://doi.org/10.1037/1082-989X.9.4.466
5. Hu, L., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
6. Kline, R.B. (2011). Principles and Practice of Structural Equation Modeling. Guilford Press.
7. Komorita, S.S., & Graham, W.K. (1965). Number of scale points and the reliability of scales. Educational and Psychological Measurement, 25(4), 987 995. https://doi.org/10.1177/001316446502500404
8. Kılıç, A.F. (2022). The effect of categories and distribution of variables on correlation coefficients. Ege Eğitim Dergisi, 23(1), 50-80. https://doi.org/10.12984/egeefd.890104

9. Kyriazos, T.A. (2018). Applied psychometrics: Sample size and sample power considerations in factor analysis (EFA, CFA) and SEM in general. Psychology, 9(8), 2207 2230. https://doi.org/10.4236/psych.2018.98126
10. Lozano, L., García-Cueto, E., & Muñiz, J. (2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology 4(2). 73 79. https://doi.org/10.1027/1614-2241.4.2.73
11. MacCallum, R.C., Widaman, K.F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99. https://doi.org/10.1037/1082-989X.4.1.84
12. Matell, M.S., & Jacoby, J. (1971). Is there an optimal number of alternatives for Likert scale items? I. Reliability and validity. Educational and Psychological Measurement, 31, 657-674. https://doi.org/10.1177/001316447103100307
13. Maydeu-Olivares, A., Fairchild, A.J., & Hall, A.G. (2017). Goodness of fit in item factor analysis: Effect of the number of response alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 24(4), 495 505. https://doi.org/10.1080/10705511.2017.1289816
14. Orçan, F. (2021). MonteCarloSEM: An R package to simulate data for SEM. International Journal of Assessment Tools in Education, 8(3), 704 713. https://doi.org/10.21449/ijate.804203
15. Orçan, F., & Yanyun, Y. (2016). A note on the use of item parceling in structural equation modeling with missing data. Journal of Measurement and Evaluation in Education and Psychology, 7(1), 59-72. https://doi.org/10.21031/epod.88204
16. Preston, C.C., & Colman, A.M. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104(1), 1–15. https://doi.org/10.1016/S0001-6918(99)00050-5
17. R Core Team. (2020). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/
18. Rhemtulla, M., Brosseau-Liard, P.É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354 373. https://doi.org/10.1037/a0029315
19. Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
20. Shi, D., Siceloff, E.R., Castellanos, R.E., Bridges, R.M., Jiang, Z., Flory, K., & Benson, K. (2021). Revisiting the effect of varying the number of response alternatives in clinical assessment: Evidence from measuring ADHD symptoms. Assessment, 28(5), 1287-1300. https://doi.org/10.1177/1073191120952885
21. Simms, L.J., Zelazny, K., Williams, T.F., & Bernstein, L. (2019). Does the number of response options matter? Psychometric perspectives using personality questionnaire data. Psychological Assessment, 31(4), 557-566. http://dx.doi.org/10.1037/pas0000648
22. Yoon, G. (2024) No one optimal way to measure people’s attitudes? Preferred length of scales in advertising research. Journal of Current Issues & Research in Advertising, 45(1), 43-70, https://doi.org/10.1080/10641734.2023.2246049
23. Wakita, T., Ueshima, N., & Noguchi, H. (2012). Psychological distance between categories in the Likert scale: Comparing different numbers of options. Educational and Psychological Measurement, 72(4), 533-546. https://doi.org/10.1177/0013164411431162