Examining the Dimensionality and Monotonicity of an Attitude Dataset based on the Item Response Theory Models

Abstract

In the current study, the factor structure of an attitude scale was analyzed by using the two different item response theory models that allow modeling non-monotonic item response curves. The current study utilized the two models to examine whether the two-factor solution of factor analysis may be caused by method effect, or by the failure of the analysis in describing and fitting the dataset because of the monotonicity assumption. This study was conducted on a dataset obtained from 355 undergraduate students who were studying at the Middle East Technical University. The data were obtained by carrying out the Attitude Scale Towards Foreign Languages as Medium of Instruction, which was developed by Kartal and Gülleroğlu (2015). The fit of the scale items to the generalized graded unfolding model was examined based on the item response curves, item parameters, item fit statistics and fit graphics. For Mokken scaling, scalability coefficients were calculated, dimensionality analyzes were conducted by using the Automated Item Selection Procedure. The monotonicity assumption was investigated based on the rest-score group methods. The results of the current study revealed that items of the attitude scale fit to the unidimensional models that do not assume monotone increasing item response curves for all items, while the factor analysis suggested a two-factor solution for the data. Researchers are recommended to utilize statistical techniques that can identify any possible violation of the monotonicity assumption and model items having non-monotonic response curves to examine dimensionality of their data.

Keywords

• Dimensionality
• Monotonocity
• Generalized Graded Unfolding Model
• Nonparametric item response model

References

1. Carter, N. T., & Dalal, D. K. (2010). An ideal point account of the JDI work satisfaction scale. Personality and Individual Differences, 49, 743-748.
2. Chernyshenko, O. S., Stark, S. E., Drasgow, F., & Roberts, J. S. (2007). Constructing personality scales under the assumptions of an ideal point response process: Toward increasing the flexibility of personality measures. Psychological Assessment, 19(1), 88-106.
3. Chernyshenko, O. S., Stark, S., Chan, K. Y., Drasgow, F., & Williams, B. (2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36(4), 523-562.
4. DiStefano, C., & Motl, R. W. (2006) Further investigating method effects associated with negatively worded items on self-report surveys. Structural Equation Modeling, 13(3), 440-464.
5. Gorsuch, R. L. (1983). Factor analysis. Saunders.
6. Gu, H., Wen, Z., & Fan, X. (2015). The impact of wording effect on reliability and validity of the Core Self-Evaluation Scale (CSES): A bi-factor perspective. Personality and Individual Differences, 83, 142-147.
7. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1985). Principles and applications of item response theory. SAGE Publications, Inc.
8. Horan, P. M., DiStefano, C., & Motl, R. W. (2003) Wording effects in self-esteem scales: Methodological artifact or response style?. Structural Equation Modeling, 10(3), 435-455.

9. Junker, B. (2000). Some topics in nonparametric and parametric IRT, with some thoughts about the future. Unpublished manuscript. Carnegie Mellon University.
10. Junker, B. W., & Sijtsma, K. (2001). Nonparametric item response theory in action: An overview of the special issue. Applied Psychological Measurement, 25(3), 211-220.
11. Ligtvoet, R., Van der Ark, L. A., Te Marvelde, J. M., & Sijtsma, K. (2010). Investigating an invariant item ordering for polytomously scored items. Educational and Psychological Measurement, 70(4), 578-595.
12. Meijer, R. R., & Egberink, I. J. (2011). Investigating invariant item ordering in personality and clinical scales: some empirical findings and a discussion. Educational Testing and Measurement, 20(10), 589-607.
13. Meijer, R. R., Tendeiro, J. N., & Wanders, R. B. (2014). The use of nonparametric item response theory to explore data quality. In Handbook of Item Response Theory Modeling (pp. 103-128). Routledge.
14. Reise, S. P., & Revicki, D. A. (2015). Handbook of item response theory modeling. Taylor & Francis Group.
15. Roberts, J. S. (1995). Item response theory approaches to attitude measurement [Doctoral dissertation, University of South Caroline, USA].
16. Roberts, J. S. (2016). Generalized graded unfolding model. W. J. van der Linden (Eds.) Handbook of item response theory volume one: Models. (pp. 369-393). Taylor & Francis Group.
17. Roberts, J. S., Donoghue, J. R., & Laughlin, J. E. (1999). Estimating parameters in the generalized graded unfolding model: Sensitivity to the prior distribution assumption and the number of quadrature points used. Paper presented at the Annual Meeting of the National Council on Measurement in Education.
18. Sijtsma, K., & Molenaar, I. W. (2002). Introduction to nonparametric item response theory (Vol. 5). Sage Publications.
19. Sijtsma, K., & Van der Ark, L. A. (2017). A tutorial on how to do a Mokken scale analysis on your test and questionnaire data. British Journal of Mathematical and Statistical Psychology, 70(1), 137-158.
20. Spector, P. E., Katwyk, P. T., Brannick, M. T., & Chen, P. Y. (1997). When two factors don’t reflect two constructs: How item characteristic can produce artifactual factors? Journal of Management, 23(5), 659-677.
21. Speer, A. B., Robie, C., & Christiansen, N. D. (2016). Effects of item type and estimation method on the accuracy of estimated personality trait scores: Polytomous item response theory models versus summated scoring. Personality and Individual Differences, 102, 41–45.
22. Stark, S. (2001). MODFIT: A computer program for model-data fit. University of Illinois at Urbana-Champaign.
23. Stevens, J. (1996). Applied multivariate statistics for the social science. Lawrence Erlbaum Associates.
24. Studts, C. R. (2008). Improving screening for externalizing behavior problems in very young children: Applications of item response theory to evaluate instruments in pediatric primary care [Doctoral dissertation, University of Louisville]. https://kb.osu.edu/
25. Supple, A. J., & Plunkett, S. W. (2011). Dimensionality and validity of the Rosenberg Self-Esteem Scale for use with Latino adolescents. Hispanic Journal of Behavioral Sciences, 33(1), 39-53.
26. Tay, L., & Drasgow, F. (2012). Theoretical, statistical, and substantive issues in the assessment of construct dimensionality: Accounting for the item response process. Organizational Research Methods, 15(3), 1-22.
27. Tendeiro, J., & Castro-Alvarez, S. (2019). GGUM: An R package for fitting the generalized graded unfolding model. Applied Psychological Measurement, 43(2), 172-173.
28. Thomas, J. M., & Oliver, A. (1999) Rosenberg's self-esteem scale: Two factors or method effects. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 84-98.
29. Van der Ark, L. A. (2007). Mokken scale analysis in R. Journal of statistical software, 20(11), 1-19.
30. Van der Linden W. J. & Hamleton, R.K. Handbook of modern item response theory (1997). Springer-Verlag.
31. Van Schuur, W. H. (2003). Mokken scale analysis: Between the Guttman scale and parametric item response theory. Political Analysis, 11(2), 139-163.
32. Van Schuur, W. H., & Kiers, H. A. L. (1994). Why factor analysis often is the incorrect model for analyzing bipolar concepts, and what model to use instead? Applied Psychological Measurement, 18(2), 97-110.
33. Wang, J., Siegal, H. A., Falck, R. S., & Carlson, R. G. (2001) Factorial structure of Rosenberg's Self-Esteem Scale among crack-cocaine drug users. Structural Equation Modeling, 8(2), 275-286.
34. Wang, Y., Kim, E. U., Dedrick, R. F., Ferron, J. M., & Tan, T. (2018). A multilevel bifactor approach to construct validation of mixed-format scales. Educational and Psychological Measurement, 78(2), 253-271.
35. Wismeijer, A. A., Sijtsma, K., van Assen, M. A., & Vingerhoets, A. J. (2008). A comparative study of the dimensionality of the self-concealment scale using principal components analysis and Mokken scale analysis. Journal of Personality Assessment, 90(4), 323-334.
36. Wouters, E, Booysen, F. L. R., Ponnet, K., & Baron, Van Loon, F. (2012). Wording effects and the factor structure of the Hospital Anxiety & Depression Scale in HIV/AIDS patients on antiretroviral treatment in South Africa. PLoS ONE, 7(4), 1-10.
37. Zijlstra, W. P., Van der Ark, L. A., & Sijtsma, K. (2011). Robust Mokken scale analysis by means of the forward search algorithm for outlier detection. Multivariate behavioral research, 46(1), 58-89.