Which scale short form development method is better? A Comparison of ACO, TS, and SCOFA

Abstract

The purpose of this study is to identify which scale short-form development method produces better findings in different factor structures. A simulation study was designed based on this purpose. Three different factor structures and three simulation conditions were selected. As the findings of this simulation study, the model-data fit and reliability coefficients were reported for each factor structure in each simulation condition. All analyses were conducted under the R environment. According to the findings of this study, the increase in the level of misspecification and the decrease in the sample size can significantly affect the model-data fit. In a situation where the factor structure of the scale is getting more and more complex, model-data fit and Omega coefficients decrease. For scales with a unidimensional factor structure, all of the scale short-form development methods are recommended. For scales with multidimensional factor structure, Ant Colony Optimization, and Stepwise Confirmatory Factor Analysis algorithms and for scales with bifactor factor structure, the ACO algorithm is recommended. When viewed from the framework of metaheuristic algorithms, it has been identified that ACO produces better findings than Tabu Search.

Keywords

• Short form
• Scale development
• Ant colony optimization
• Tabu search
• Stepwise confirmatory factor analysis

References

1. Anastasi, A. (1982). Psychological Testing (5th ed.). Macmillan.
2. Batley, R.M., & Boss, M.W. (1993). The effects on parameter estimation of correlated dimensions and a distribution-restricted trait in a multidimensional item response model. Applied Psychological Measurement, 17(2), 131 141. https://doi.org/10.1177/014662169301700203
3. Cayanus, J.L., & Martin, M.M. (2004). An instructor self‐disclosure scale. Communication Research Reports, 21(3), 252-263. https://doi.org/10.1080/08824090409359987
4. Colorni, A., Dorigo, M., & Maniezzo, V. (1991). Distributed optimization by ant colonies. In: Varela, F. and Bourgine, P., Eds., Proceedings of the European Conference on Artificial Life, ECAL’91, Paris, Elsevier Publishing, Amsterdam, 134-142.
5. Ebesutani, C., McLeish, A.C., Luberto, C.M., Young, J., & Maack, D.J. (2014). A bifactor model of anxiety sensitivity: Analysis of the Anxiety Sensitivity Index-3. Journal of Psychopathology and Behavioral Assessment, 36(3), 452 464. https://doi.org/10.1007/s10862-013-9400-3
6. French, B.F., & Finch, W.H. (2011). Model misspecification and invariance testing using confirmatory factor analytic procedures. The Journal of Experimental Education, 79(4), 404-428. https://doi.org/10.1080/00220973.2010.517811
7. Gatignon, H. (2010). Confirmatory Factor Analysis. In Statistical Analysis of Management Data (pp. 59-122). Springer. https://doi.org/10.1007/978-1-4419-1270-1_4
8. Hu, L.T., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118

9. Janssen, A.B., Schultze, M., & Grötsch, A. (2017). Following the ants: Development of short scales for proactive personality and supervisor support by Ant Colony Optimization. European Journal of Psychological Assessment, 33(6), 409. https://doi.org/10.1027/1015-5759/a000299
10. Jiang, S., Wang, C., & Weiss, D.J. (2016). Sample size requirements for estimation of item parameters in the multidimensional graded response model. Frontiers in Psychology, 7(Article:109), 1-10. https://doi.org/10.3389/fpsyg.2016.00109
11. Jorgensen, T.D., Pornprasertmanit, S., Schoemann, A. M., Rosseel, Y., Miller, P., Quick, C., ..., & Enders, C. (2016). semTools: Useful Tools for Structural Equation Modeling. R package version 0.5 4. Retrieved from https://cran.r project.org/web/packages/semTools/index.html
12. Kleka, P., & Soroko, E. (2018). How to avoid the sins of questionnaires abridgement?. Survey Research Methods, 12(2), 147-160. https://doi.org/10.31234/osf.io/8jg9u
13. Kruyen, P.M., Emons, W.H., & Sijtsma, K. (2013). On the shortcomings of shortened tests: A literature review. International Journal of Testing, 13(3), 223 248. https://doi.org/10.1080/15305058.2012.703734
14. LaNoue, M., Harvey, A., Mautner, D., Ku, B., & Scott, K. (2015). Confirmatory factor analysis and invariance testing between Blacks and Whites of the Multidimensional Health Locus of Control scale. Health Psychology Open, 2(2), 1 16. https://doi.org/10.1177/2055102915615045
15. Leite, W.L., Huang, I.-C., & Marcoulides, G. A. (2008). Item selection for the development of short forms of scales using an Ant Colony Optimization Algorithm. Multivariate Behavioral Research, 43, 411–431. https://doi.org/10.1080/00273170802285743
16. Marcoulides, K.M., & Falk, C. (2018). Model specification searches in structural equation modeling with R. Structural Equation Modeling, 25(3), 484 491. https://doi.org/10.1080/10705511.2017.1409074
17. Nunnally, J.C. (1978). Psychometric Theory (2nd ed.). McGraw-Hill.
18. Olaru, G., Witthöft, M., & Wilhelm, O. (2015). Methods matter: Testing competing models for designing short-scale big-five assessments. Journal of Research in Personality, 59, 56-68. https://doi.org/10.1016/j.jrp.2015.09.001
19. Raborn, A.W., & Leite, W.L. (2018). ShortForm: An R package to select scale short forms with the ant colony optimization algorithm. Applied psychological measurement, 42(6), 516. https://doi.org/10.1177/0146621617752993
20. Raborn, A.W., Leite, W.L., & Marcoulides, K.M. (2020). A comparison of metaheuristic optimization algorithms for scale short-form development. Educational and Psychological Measurement, 80(5), 910 931. https://doi.org/10.1177/0013164420906600
21. Reise, S.P. (2012). The rediscovery of bifactor measurement models. Multivariate Behav. Res. 47, 667–696. https://doi.org/10.1080/00273171.2012.715555
22. Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling and more. Version 0.5 12 (BETA). Journal of Statistical Software, 48(2), 1 36. https://doi.org/10.18637/jss.v048.i02
23. Schroeders, U., Wilhelm, O., & Olaru, G. (2016). Meta-heuristics in short scale construction: Ant colony optimization and genetic algorithm. PLoS One, 11(11), 1-19. https://doi.org/10.1371/journal.pone.0167110
24. Singh, K., Junnarkar, M., & Kaur, J. (2016). Measures of Positive Psychology: Development and Validation. Springer.
25. Van Abswoude, A.A., van der Ark, L.A., & Sijtsma, K. (2004b). A comparative study of test data dimensionality assessment procedures under nonparametric IRT models. Applied Psychological Measurement, 28(1), 3-24. https://doi.org/10.1177/0146621603259277
26. Van Abswoude, A.A., Vermunt, J.K., Hemker, B.T., & van der Ark, L.A. (2004a). Mokken scale analysis using hierarchical clustering procedures. Applied Psychological Measurement, 28(5), 332-354. https://doi.org/10.1177/0146621604265510
27. Yang, Y., & Liang, X. (2013). Confirmatory factor analysis under violations of distributional and structural assumptions. International Journal of Quantitative Research in Education, 1(1), 61-84. https://doi.org/10.1504/ijqre.2013.055642