The comparison of the dimensionality results provided by the automated item selection procedure and DETECT analysis

Abstract

The dimensionality is one of the most investigated concepts in the psychological assessment, and there are many ways to determine the dimensionality of a measured construct. The Automated Item Selection Procedure (AISP) and the DETECT are non-parametric methods aiming to determine the factorial structure of a data set. In the current study, dimensionality results provided by the two methods were compared based on the original factorial structure defined by the scale developers. For the comparison of the two methods, the data was obtained by implementing a scale measuring academic dishonesty levels of bachelor students. The scale was conducted on junior students studying at a public and a private university. The dataset was analyzed by using the AISP and DETECT analyses. The “mokken” and “sirt” packages on the R program were utilized for the AISP and DETECT analyses, respectively. The similarities and differences between the findings provided by the methods were analyzed depending on the original factor structure of the scale verified by the scale developers.

Keywords

• Dimensionality
• Mokken scaling DETECT
• Automated item selection procedure
• Non parametric item response theory

References

1. Ackerman, T.A., Gierl, M.A., & Walker, C.M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22(1), 37-53. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
2. Antino, M., Alvarado, J.M., Asún, R.A., & Bliese, P. (2020). Rethinking the exploration of dichotomous data: Mokken scale analysis versus factorial analysis. Sociological Methods Research, 49(4), 839-867. https://doi.org/10.1177/0049124118769090
3. Cavalini, P.M. (1992). It’s an ill wind that brings no good. Studies on odour annoyance and the dispersion of odorant concentrations from industries [Unpublished doctoral dissertation]. University of Groningen, The Netherlands.
4. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Cengage Learning.
5. Finch, H. (2010). Item parameter estimation for the MIRT model bias and precision of confirmatory factor analysis based models. Applied Psychological Measurement 34(1), 10 26. https://doi.org/10.1177/0146621609336112
6. Finch, H. (2011). Multidimensional item response theory parameter estimation with non-simple structure items. Applied Psychological Measurement, 35(1), 67 82. https://doi.org/10.1177/0146621610367787
7. Guttman, L. (1944). A basis for scaling qualitative data. American Sociological Review, 9(1), 255-282.
8. Guttman, L. (1950). The basis for scalogram analysis. In S.A. Stouffer, L. Guttman, E.A. Suchman, P.F. Lazarsfeld, S.A. Star, & J.A. Clausen (Eds.), Measurement and prediction (pp. 60-90). Princeton University Press.

9. Hattie, J. (1985). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9(2), 139 164. https://doi.org/10.1177/014662168500900204
10. Hattie, J., Krakowski, K., Jane Rogers, H., & Swaminathan, H. (1996). An assessment of Stout's index of essential unidimensionality. Applied Psychological Measurement, 20(1), 1 14. https://doi.org/10.1177/014662169602000101
11. Hemker, B.T., & Sijtsma, K. (1993). A practical comparison between the weighted and the unweighted scalability coefficient of the Mokken model. Kwantitatieve Methoden, 14(44), 59-73.
12. Hemker, B.T., Sijtsma, K., & Molenaar, I.W. (1995). Selection of unidimensional scales from a multidimensional item bank in the polytomous Mokken IRT model. Applied Psychological Measurement, 19(4), 337 352. https://doi.org/10.1177/014662169501900404
13. Junker, B.W. (1993). Conditional association, essential independence and monotone unidimensional item response models. The Annals of Statistics, 21(1), 1359 1378. https://doi.org/10.1214/aos/1176349262
14. Kane, M.T. (2006). Validation. Educational Measurement, 4(2), 17 64. https://doi.org/10.1111/j.1745-3992.1985.tb00874.x
15. Kim, H.R. (1994). New techniques for the dimensionality assessment of standardized test data [Doctoral dissertation, University of Illinois at Urbana Champaign]. ProQuest Dissertations and Theses Global.
16. Kuijpers, R.E., van der Ark, L.A. & Croon, M.A. (2013). Standard errors and confidence intervals for scalability coefficients in Mokken scale analysis using marginal models. Sociological Methodology, 43(1), 42-69. https://doi.org/10.1177/0081175013481958
17. Lissitz, R.W. (Ed.). (2009). The concept of validity: Revisions, new directions and applications. Information Age Publishing.
18. Loevinger, J. (1948). The technique of homogeneous tests compared with some aspects of scale analysis and factor analysis. Psychological Bulletin, 45(1), 507 530. https://doi.org/10.1037/h0055827
19. Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Addison Wesley.
20. Meijer, R.R., Sijtsma, K., & Smid, N. (1990). Theoretical and empirical comparison of the Mokken and the Rasch approach to IRT. Applied Psychological Measurement, 14(1), 283-298. https://doi.org/10.1177/014662169001400306
21. Messick, S. (1975). The standard problem: Meaning and values in measurement and evaluation. American Psychologist, 30(10), 955 966. https://doi.org/10.1037/0003 066X.30.10.955
22. Mokken, R.J. (1971). A theory and procedure of scale analysis. De Gruyter.
23. Mokken, R.J., & Lewis, C. (1982). A nonparametric approach to the analysis of dichotomous item responses. Applied Psychological Measurement, 6(1), 417 430. https://doi.org/10.1177/014662168200600404
24. Mokken, R.J., Lewis, C., & Sijtsma, K. (1986). Rejoinder to The Mokken scale: A critical discussion. Applied Psychological Measurement, 10(1), 279 285. https://doi.org/10.1177/014662168601000306
25. Molenaar, I.W. (1982). Mokken scaling revisited. Kwantitatieve Methoden, 3(8), 145-164. Molenaar, I.W. (1991). A weighted Loevinger H-coefficient extending Mokken scaling to multicategory items. Kwantitatieve Methoden, 12(37), 97-117.
26. Molenaar, I.W. (in press). Nonparametric models for polytomous responses. In W.J. van der Linden, & R.K. Hambleton (Eds.), Handbook of modern psychometrics (pp. 361-373). Springer.
27. Molenaar, I.W., Debets, P., Sijtsma, K., & Hemker, B. T. (1994). User’s manual for the computer program MSP (Ver. 3.0). ProGAMMA.
28. Mroch, A.A., & Bolt, D.M. (2006). A simulation comparison of parametric and nonparametric dimensionality detection procedures. Applied Measurement in Education, 19(1), 67-91. https://doi.org/10.1207/s15324818ame1901_4
29. Nandakumar, R., & Ackerman, T. (2004). Test modeling. In D. Kaplan (Ed.), The SAGE handbook of quantitative methodology for the social sciences (pp. 93 107). SAGE Publications.
30. Reise, S.P., & Waller, N.G. (2003). How many IRT parameters does it take to model psychopathology items? Psychological Methods, 8(2), 164 184. https://doi.org/10.1037/1082-989X.8.2.164
31. Roussos, L.A., & Özbek, Ö.Y. (2006). Formulation of the DETECT population parameter and evaluation of detect estimator bias. Journal of Educational Measurement, 43(3), 215 243. https://doi.org/10.1111/j.1745-3984.2006.00014.x
32. Roussos, L.A., Stout, W.F., & Marden, J.I. (1998). Using new proximity measures with hierarchical cluster analysis to detect multidimensionality. Journal of Educational Measurement, 35(1), 1-30. https://doi.org/10.1111/j.1745-3984.1998.tb00525.x
33. Roznowski, M., Humphreys, L.G., & Davey, T. (1994). A simplex fitting approach to dimensionality assessment of binary data matrices. Educational and Psychological Measurement, 54(2), 263-283. https://doi.org/10.1177/0013164494054002002
34. Sick, J. (2010). Assumptions and requirements of Rasch measurement. SHIKEN: JALT Testing & Evaluation SIG Newsletter, 14(2), 23-29.
35. Sijtsma, K., & Molenaar, I.W. (2002). Introduction to nonparametric item response theory. SAGE Publications.
36. Sireci, S.G. (2009). Packing and unpacking sources of validity evidence: History repeats itself again. In R.W., Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 19–37). IAP Information Age Publishing.
37. Slocum-Gori, S.L., & Zumbo, B.D. (2011). Assessing the unidimensionality of psychological scales: Using multiple criteria from factor analysis. Social Indicators Research, 102(3), 443-461. https://doi.org/10.1007/s11205-010-9682-8
38. Stochl J, Jones P.B., & Croudace, T.J. (2012). Mokken scale analysis of mental health and well-being questionnaire item responses: A non-parametric IRT method in empirical research for applied health researchers. BMC Medical Research Methodology, 12(1), 1 16. https://doi.org/10.1186/1471-2288-12-74
39. Stocking, M.L., Swanson, L., & Pearlman, M. (1991). Automated item selection using item response theory. Educational Testing Service.
40. Stocking, M.L., Swanson, L., & Pearlman, M. (1993). Application of an automated item selection method to real data. Applied Psychological Measurement, 17(2), 167 176. https://doi.org/10.1177/014662169301700206
41. Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52(4), 589-617. https://doi.org/10.1007/BF02294821
42. Stout, W. F. (1990). A new item response theory modelling approach with applications to unidimensionality assessment and ability estimation. Psychometrika, 49(1), 293 325. https://doi.org/10.1007/BF02295289
43. Stout, W. (2002). Psychometrics: From practice to theory and back. Psychometrika, 67(4), 485-518. https://doi.org/10.1007/BF02295128
44. Stout, W., Froelich, A.G., & Gao, F. (2001). Using resampling methods to produce an improved DIMTEST procedure. In A. Boomsma, M.A.J. van Duijn, & T.A.B., Snijders (Eds.), Essays on item response theory (pp. 357-375). Springer.
45. Stout, W., Habing, B., Douglas, J., Kim, H.R., Roussos, L., & Zhang, J. (1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 20(4), 331 354. https://doi.org/10.1177/014662169602000403
46. Stout, W., Nandakumar, R., & Habing, B. (1996). Analysis of latent dimensionality of dichotomously and polytomously scored test data. Psychometrika, 23(1), 37 65. https://doi.org/10.2333/bhmk.23.37
47. Tate, R. (2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27(1),159-203.
48. van Abswoude, A.A., van der Ark, L.A., & Sijtsma, K. (2004). A comparative study of test data dimensionality assessment procedures under nonparametric IRT models. Applied Psychological Measurement, 28(1), 3-24. https://doi.org/10.1177/0146621603259277
49. van der Ark, L.A. (2007) Mokken scale analysis in R. Journal of Statistical Software, 20(11), 1-19. https://doi.org/10.18637/jss.v020.i11
50. van der Eijk, C. & Jonathan, R. (2015). Risky business: Factor analysis of survey data-assessing the probability of incorrect dimensionalisation. PLoS One, 10(3), 1 35. https://doi.org/10.1371/journal.pone.0118900
51. Wismeijer, A.A.J, Sijtsma, K., van Assen M.A.L.M & Vingerhoets, J.J.M. (2008). A comparative study of the dimensionality of the self-concealment scale using principal components analysis and mokken scale analysis. Journal of Personality Assessment, 90(4), 323-334. https://doi.org/10.1080/00223890802107875
52. Yu, C.H., Osborn Popp, S., DiGangi, S., & Jannasch Pennell, A. (2007). Assessing unidimensionality: A comparison of Rasch modeling, parallel analysis, and TETRAD. Practical Assessment, Research, and Evaluation, 12(1), 1-19. https://doi.org/10.7275/q7g0-vt50
53. Yu, F., & Nandakumar, R. (2001). Poly Detect for quantifying the degree of multidimensionality of item response data. Journal of Educational Measurement, 38(2), 99 120. https://doi.org/10.1111/j.1745-3984.2001.tb01118.x
54. Zhang, J., & Stout, W. (1999). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64(2), 213 249. https://doi.org/10.1007/ß
55. Zumbo, B.D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R.W., Lissitz (Ed.), The Concept of Validity: Revisions, New Directions and Applications. IAP Information Age Publishing.