A Comparison of type I error and power rates in procedures used determining test dimensionality

Abstract

The purpose of this study was to investigate the Type I Error findings and power rates of the methods used to determine dimensionality in unidimensional and bidimensional psychological constructs for various conditions (characteristic of the distribution, sample size, length of the test, and interdimensional correlation) and to examine the joint effect of the conditions (effect of the interaction of conditions) as well as the main effect of each condition. The simulative data were generated for the study using the SAS program. Within the scope of the study, the data were analyzed using the DIMTEST T statistic and the Dimensionality DETECT IDN index, which is one of the non-parametric methods. The Nonlinear Factor Analysis (NOHARM) method was preferred from among parametric methods. As a result of the study, it was noted that the most consistent results in making the unidimensionality decisions belong to the Nonlinear Factor Analysis method showing standard normal distribution according to the shape of the distribution. When the power study results were examined, it was noted that the DIMTEST T statistic gave more accurate results in conditions with large samples, consisting of data with standard normal distribution. On the other hand, while results of the DETECT IDN index and Nonlinear factor analysis were more internally consistent, it was noted that in conditions where the sample size was 1000 and above, the DIMTEST T statistic also made the right decisions in determining dimensionality.

Keywords

• Dimensionality
• Construct validity
• DIMTEST T statistic
• DETECT
• Nonlinear factor analysis

References

1. Ackerman, T.A. (1994). Using multidimensional item response theory to understand what items and tests are measuring. Applied Measurement in Education, 7(4), 255-278. https://doi.org/10.1207/s15324818ame0704_1
2. Ackerman, T.A., Gierl, M.J., & Walker, C.M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22(3), 37-51. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
3. Davey, T., Nering M.L., & Thompson, T. (1997). Realistic simulation of item response data. ACT Research Report Series, 97-4. https://files.eric.ed.gov/fulltext/ED414297.pdf
4. Embretson, S.E., & Reise, S. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates.
5. Finch, H., & Habing, B. (2007). Performance of DIMTEST- and NOHARMbased statistics for testing unidimensionality. Applied Psychological Measurement, 31, 292-307. https://doi.org/10.1177/0146621606294490
6. Fleisman, A.I. (1978). A method for simulating non-normal distributions. Psychometrika, 43, 521-532. https://doi.org/10.1007/BF02293811
7. Froelich, A.G., & Habing, B. (2008). Conditional covariance-based subtest selection for DIMTEST. Applied Psychological Measurement, 32, 138 155. https://doi.org/10.1177/0146621607300421
8. Gessaroli, M.E., & De Champlain, A.F. (1996). Using an approximate chi-square statistic to test the number of dimensions underlying the responses to a set of items. Journal of Educational Measurement, 33, 157 179. https://doi.org/10.1111/j.1745 3984.1996.tb00487.x Göçer Şahin, S. (2016). Examining parameter estimation when treating semi mixed multidimensional constructs as unidimensional [Unpublished doctoral dissertation]. Hacettepe University

9. Hattie, J. (1985). Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9(8), 139 – 145. http://dx.doi.org/10.1177/014662168500900204
10. Hattie, J., Krakowski, K., Rogers, H.J., & Swaminathan, H. (1996). An assessment of Stout’s index of essential dimensionality. Applied Psychological Measurement, 20, 1-14. https://doi.org/10.1177%2F014662169602000101
11. Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods 6(1), 53-60. https://doi.org/10.21427/D7CF7R
12. Hu, L-T., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. https://doi.org/10.1080/10705519909540118
13. Kaya, K.Ö., & Kelecioğlu, H. (2016). The effect of sample size on parametric and nonparametric factor analytical methods. Educational Sciences: Theory & Practice. 16(1), 153-171. http://dx.doi.org/10.12738/estp.2016.1.0220
14. Kim, H.R. (1994). New techniques for dimensionality assessment of standardized test data. [Unpublished doctoral dissertation]. University of Illinois at Urbana-Champaign, Department of Statistics.
15. Ledasma, R.D., & Valero-Mora, P. (2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out Parallel Analysis. Practical Assessment, Research & Evaluation, 12 (2). https://doi.org/10.7275/wjnc-nm63
16. Mroch, A.A., & Bolt, D.M. (2006). A simulation comparison of parametric and nonparametric dimensionality detection procedures. Applied Measurement in Education, 19 (1), 67-91. https://doi.org/10.1207/s15324818ame1901_4
17. Nandakumar, R., & Stout, W. (1993). Refinement of Stout’s procedure for assessing latent trait unidimensionality. Journal of Educational Statistics, 18(1), 41 68. https://psycnet.apa.org/doi/10.2307/1165182
18. Özbek Baştuğ, Ö.Y. (2012). Assessment of Dimensionality in Social Science Subtest. Educational Sciences: Theory & Practice. 12(1), Winter: 382-385.
19. Reichenberg, R.E. (2013). A comparison of DIMTEST and generalized dimensionality discrepancy approaches to assessing dimensionality in item response theory [M.S. dissertation, Arizona State University, Arizona]. https://doi.org/10.3102%2F10769986018001041
20. Reckase, M.D. (2009). Multidimensional item response theory. Springer Dordrecht Heidelberg.
21. Roussos, L.A., & Özbek, O.Y. (2006). Formulation of the DETECT population parameter and evaluation of DETECT estimator bias. Journal of Educational Measurement, 43, 215-243. https://doi.org/10.1111/j.1745-3984.2006.00014.x
22. Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617. https://doi.org/10.1007/BF02294821
23. Stout, W., Habing, B., Douglas, J., Kim, H.R., Roussos, L., & Zhang J. (1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 19, 331-354. https://doi.org/10.1177%2F014662169602000403
24. Sünbül, Ö. (2011). Çeşitli boyutluluk özelliklerine sahip yapılarda, madde parametrelerinin değişmezliğinin klasik test teorisi, tek boyutlu madde tepki kuramı ve çok boyutlu madde tepki kuramı çerçevesinde incelenmesi [Unpublished doctoral dissertation]. Mersin University.
25. Sünbül, Ö., & Seo, M. (2012). Performance of test statistics for verifying unidimensionality, [Conference presentation abstract]. 2012 Annual Meeting, April 12-16, Vancouver, British Columbia, CANADA
26. Svetina, D. (2011). Assessing dimensionality in complex data structures: A performance comparison of DETECT and NOHARM procedures [Unpublished doctoral dissertation]. Arizona State University
27. Svetina, D., & Levy, R. (2014). A framework for dimensionality assessment for multidimensional item response models. Educational Assessment, 19(1), 35-57. https://doi.org/10.1080/10627197.2014.869450
28. Tate, R. (2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27, 159-203. https://doi.org/10.1177/0146621603027003001
29. Touron, J., Lizasoain, L., & Joaristi, L. (2012). Assessing the unidimensionality of the School and College Ability Test (SCAT, Spanish version) using non-parametric methods based on item response theory. High Ability Studies. 23(2), 183 202. https://doi.org/10.1080/13598139.2012.735401
30. Zhang, B. (2008). Application of unidimensional item response models to tests with items sensitive to secondary dimension. The Journal of Experimental Education, 77 (2), 147-166. https://doi.org/10.3200/JEXE.77.2.147-166