Item Removal Strategies Conducted in Exploratory Factor Analysis: A Comparative Study

Abstract

The aim of this study is to examine how the practice of different item removal strategies during exploratory factor analysis (EFA) phase of scale development change the number of factors, factor loadings, explained variance ratio, and reliability values (α and ω) explained. In the study, data obtained from 379 university students were used for the development of a 46-item scale. As the first item removal strategy, crossloading items on two factors and where the difference between factor loadings was less than .10 were identified. Then, items were removed one by one, starting with the item with the least difference between the loadings on the factors. As the second strategy, the items that loaded on two factors and where the difference between factor loadings was less than .10 were found, and these items were removed from the scale as a whole. As the third strategy, the items that gave high loading on more than two factors and where the difference between these factors was less than .10 were identified. The item removal process was started with these items. The study results show that the factor numbers obtained using three different strategies during the item removal process of EFA were the same; however, the number of items on the scale, the explained variance ratio, and the total scale, and reliability values differed. Furthermore, the items in the factors were not all the same. The study results underscore the importance of theoretical competence in the scale development process.

Keywords

• Exploratory Factor Analysis
• Item removal strategies
• Crossloading items
• Construct validity
• Scale development.

Öz

The aim of this study is to examine how the practice of different item removal strategies during exploratory factor analysis (EFA) phase of scale development change the number of factors, factor loadings, explained variance ratio, and reliability values (α and ω) explained. In the study, data obtained from 379 university students were used for the development of a 46-item scale. As the first item removal strategy, crossloading items on two factors and where the difference between factor loadings was less than .10 were identified. Then, items were removed one by one, starting with the item with the least difference between the loadings on the factors. As the second strategy, the items that loaded on two factors and where the difference between factor loadings was less than .10 were found, and these items were removed from the scale as a whole. As the third strategy, the items that gave high loading on more than two factors and where the difference between these factors was less than .10 were identified. The item removal process was started with these items. The study results show that the factor numbers obtained using three different strategies during the item removal process of EFA were the same; however, the number of items on the scale, the explained variance ratio, and the total scale, and reliability values differed. Furthermore, the items in the factors were not all the same. The study results underscore the importance of theoretical competence in the scale development process.

Anahtar Kelimeler

• Exploratory Factor Analysis
• Item removal strategies
• Crossloading items
• Construct validity
• Scale development.

References

1. Albayrak, A.S. (2006). Uygulamalı çok değişkenli istatistik teknikleri [Applied multivariate statistical techniques]. Asil Yayın Dağıtım.
2. Albayrak, A.S. (2005). Çoklu doğrusal bağlantı halinde en küçük kareler tekniğinin alternatifi yanlı tahmin teknikleri ve bir uygulama [Alternative to the minimum square technique: a multi-linear connection balanced estimating techniques and an application]. Zonguldak Karaelmas Üniversitesi Sosyal Bilimler Dergisi, 1(1), 105 126.
3. Bartlett, M.S. (1950). Tests of significance in factor analysis. British Journal of Psychology, 3, 77-85.https://doi.org/10.1111/j.2044-8317.1950.tb00285.x
4. Basto, M., & Pereira, J.M. (2012). An SPSS R-Menu for ordinal factor analysis. Journal of statistical software, 46(4), 1-29. https://doi.org/10.18637/jss.v046.i04
5. Bornstein, R.F. (1996). Face validity in psychological assessment: Implications for a unified model of validity. American Psychologist, 51(9), 983-984. https://doi.org/10.1037/0003-066X.51.9.983
6. Brown, J.D. (2009). Statistics Corner Questions and answers about language testing statistics: Principal components analysis and exploratory factor analysis, In. Definitions, differences, and choices. Shiken: JALT Testing & Evaluation SIG Newsletter, 13(1), 19 - 23. https://hosted.jalt.org/test/PDF/Brown30.pdf
7. Büyüköztürk, Ş. (2007). Veri Analizi El Kitabı [Data analysis handbook for social sciences]. Ankara: Pegem Yayınları.
8. Bryman, A., & Cramer, D. (2011). Quantitative data analysis with IBM SPSS 17, 18 and 19: A guide for social scientists. Routledge-Cavendish/Taylor & Francis Group.

9. Can, A. (2016). SPSS ile bilimsel araştırma sürecinde nicel veri analizi [Quantitative data analysis in the process of scientific research with SPSS]. Ankara: Pegem Akademi.
10. Cattell, R.B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1(2), 245-276. https://doi.org/10.1207/s15327906mbr0102_10
11. Clark, L.A., & Watson, D. (2016). Constructing validity: Basic issues in objective scale development. In A. E. Kazdin (Ed.), Methodological issues and strategies in clinical research (pp. 187 203). American Psychological Association. https://doi.org/10.1037/14805-012
12. Comrey, A.L. (1962). The minimum residual method of factor analysis. Psychological Reports, 11(1), 15-18. https://doi.org/10.2466/pr0.1962.11.1.15
13. Comrey, A.L., & Lee, H.B. (1973). A first course in factor analysis. Academic Press.
14. Costello, A.B., & Osborne, J. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical assessment, research, and evaluation, 10(7), 1-9. https://doi.org/10.7275/jyj1-4868.
15. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. CBS College Publishing.
16. Cronbach, L.J., & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957.
17. Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk, Ş. (2010). Sosyal bilimler için çok değişkenli istatistik [Multivariate statistics for social sciences]. Pegem Akademi.
18. Erkuş, A. (2012). Psikolojide ölçme ve ölçek geliştirme [Measurement and scale development in psychology]. Pegem Akademi Yayınları.
19. Erkuş, A., Sünbül, Ö., Sünbül, S Ö., Yormaz, S., & Aşiret, S. (2017). Psikolojide ölçme ve ölçek geliştirme II [Measurement and scale development in psychology II]. Pegem Akademi.
20. Ekström, J. (2011). A Generalized Definition of the Polychoric Correlation Coefficient. UCLA: Department of Statistics, UCLA. https://escholarship.org/uc/item/583610fv
21. Field, A. (2005). Discovering statistics using SPSS. (2nd ed.). London: Sage
22. Finney, S.J., & DiStefano, C. (2013). Nonnormal and categorical data in structural equation modeling. G. R. Hancock ve R. O. Mueller (Ed.), Structural equation modeling: A second course (2nd ed., 439– 492). Charlotte.
23. Ford, J.K., MacCallum, R.C., & Tait, M. (1986). The Application of exploratory factor analysis in applied psychology: A Critical review and analysis. Personnel Psychology, 39(2), 291-314. https://doi.org/10.1111/j.1744-6570.1986.tb00583.x
24. Gorsuch, R.L. (1983). Factor analysis (2nd ed.). Hillside, NJ: Lawrence Erlbaum Associates
25. Hauben, M., Hung, E., & Hsieh, W.Y. (2017). An exploratory factor analysis of the spontaneous reporting of severe cutaneous adverse reactions. Therapeutic advances in drug safety, 8(1), 4-16. https://doi.org/10.1177/2042098616670799
26. Hayton, J.C., Allen, D.G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191-205. https://doi.org/10.1177/1094428104263675
27. Henson, R.K., & Roberts, J.K. (2006). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Educational and Psychological Measurement, 66(3), 393-416. https://doi.org/10.1177/0013164405282485
28. Horn, J.L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179-185. https://doi.org/10.1007/BF02289447
29. Johnson, R.A., & Wichern, D.W. (2002). Applied multivariate statistical analysis. Upper Saddle River.
30. Kass, R.A., & Tinsley, H.E.A. (1979). Factor analysis. Journal of Leisure Research, 11, 120-138. https://doi.org/10.1080/00222216.1979.11969385
31. Kerlinger, F.N. (1979). Behavioral research: A conceptual approach. Rinehart & Winston.
32. Kline, P. (1994). An easy guide to factor analysis. Routledge.
33. Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd Edition). The Guilford Press.
34. Leech, N.L., Barrett, K.C., & Morgan, G.A. (2005) SPSS for Intermediate Statistics, Use and Interpretation. 2nd Edition. Lawrence Erlbaum.
35. Loevinger, J. (1957). Objective tests as instruments of psychological theory. Psychological Reports, 3(3), 635–694. https://doi.org/10.2466/pr0.1957.3.3.635
36. MacCallum, R.C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84-99. https://doi.org/10.1037/1082-989X.4.1.84
37. Mardia, K.V. (1970). Measures of multivariate skewnees and kurtosis with applications. Biometrika, 57(3), 519-530. https://doi.org/10.2307/2334770
38. Mertler, C.A., & Vannatta, R.A. (2005). Advanced and multivariate statistical methods: Practical application and interpretation (3th ed.). Pyrczak Publishing.
39. Messick, S. (1981). Evidence and ethics in the evaluation of tests 1. ETS Research Report Series, 1981(1), 1-41. https://doi.org/10.1002/j.2333-8504.1981.tb01244.x
40. Murphy, K.R., & Davidshofer, C.O. (2005). Psychological testing: principles and applications. Pearson Education International.
41. Netemeyer, R.G., Bearden, W.O., & Sharma, S. (2003). Scaling procedures. SAGE Publications, Inc.
42. Nunally, J.C. (1978). Psychometric theory. McGraw Hill.
43. Özgüven, E. (1994). Psikolojik testler [Psychological tests]. Yeni Doğuş Matbaası.
44. Park, H.S., Dailey, R., & Lemus, D. (2002). The use of exploratory factor analysis and principal components analysis in communication research. Human Communication Research, 28(4), 562-577. https://doi.org/10.1111/j.1468-2958.2002.tb00824.x
45. Pett, M.A., Lackey, N.R., & Sullivan, J.J. (2003). Making sense of factor analysis: The use of factor analysis for instrument development in health care research. SAGE.
46. Raubenheimer, J. (2004). An item selection procedure to maximize scale reliability and validity. SA Journal of Industrial Psychology, 30(4), 59 64. https://doi.org/10.4102/sajip.v30i4.168
47. Samuels, P. (2017). Advice on Exploratory Factor Analysis. Technical Report. Centre for Academic Success, Birmingham City University.
48. Scherer, R.F., Luther, D.C., Wiebe, F.A., & Adams, J.S. (1988). Dimensionality of coping: Factor stability using the ways of coping questionnaire. Psychological Reports, 62(3), 763-770. https://doi.org/10.2466/pr0.1988.62.3.763
49. Sarstedt M., & Mooi E. (2014). Factor analysis. In: A concise guide to market research. springer texts in business and economics. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-53965-7_8
50. Spicer J. (2005). Making sense of multivariate data analysis: An Intuitive approach. SAGE.
51. Stapleton, C.D. (1997). Basic concepts and procedures of confirmatory factor analysis [Paper presentation]. The Annual Meeting of the South West Educational Research Association. Austin.
52. Stewens, J. (1996). Appied multivariate statistics for the social science (Third Edition). Lawrence Erlbaum Associates.
53. Şencan, H. (2005). Sosyal ve davranışsal ölçümlerde güvenirlik ve geçerlilik [Reliability and validity in social and behavioral assessments]. Seçkin Yayıncılık.
54. Tabachnick, B.G., & Fidell, L.S. (1996). Using multivariate statistics. Harper Collins.
55. Tabachnick, B.G., & Fidell, L.S. (2001). Using multivariate statistics. 4th Edition, Allyn and Bacon. MA.
56. Tavşancıl E. (2002). Tutumların ölçülmesi ve SPSS ile veri analizi [Measurement of attitudes and data analysis with SPSS]. Nobel Yayınevi.
57. Velicer, W.F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41(3), 321-327. https://doi.org/10.1007/BF02293557
58. Ziegler, M. (2014). Comments on item selection procedures. European Journal of Psychological Assessment, 30(1), 1-2. https://doi.org/10.1027/1015-5759/a000196