A Comparison of type I error and power rates in procedures used determining test dimensionality

Year 2022, Volume: 9 Issue: 3, 697 - 712, 30.09.2022

Gül Güler , Nükhet Çıkrıkçı

https://doi.org/10.21449/ijate.1059628

Abstract

The purpose of this study was to investigate the Type I Error findings and power rates of the methods used to determine dimensionality in unidimensional and bidimensional psychological constructs for various conditions (characteristic of the distribution, sample size, length of the test, and interdimensional correlation) and to examine the joint effect of the conditions (effect of the interaction of conditions) as well as the main effect of each condition. The simulative data were generated for the study using the SAS program. Within the scope of the study, the data were analyzed using the DIMTEST T statistic and the Dimensionality DETECT IDN index, which is one of the non-parametric methods. The Nonlinear Factor Analysis (NOHARM) method was preferred from among parametric methods. As a result of the study, it was noted that the most consistent results in making the unidimensionality decisions belong to the Nonlinear Factor Analysis method showing standard normal distribution according to the shape of the distribution. When the power study results were examined, it was noted that the DIMTEST T statistic gave more accurate results in conditions with large samples, consisting of data with standard normal distribution. On the other hand, while results of the DETECT IDN index and Nonlinear factor analysis were more internally consistent, it was noted that in conditions where the sample size was 1000 and above, the DIMTEST T statistic also made the right decisions in determining dimensionality.

Keywords

Dimensionality, Construct validity, DIMTEST T statistic, DETECT, Nonlinear factor analysis

References

• Ackerman, T.A. (1994). Using multidimensional item response theory to understand what items and tests are measuring. Applied Measurement in Education, 7(4), 255-278. https://doi.org/10.1207/s15324818ame0704_1
• Ackerman, T.A., Gierl, M.J., & Walker, C.M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22(3), 37-51. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
• Davey, T., Nering M.L., & Thompson, T. (1997). Realistic simulation of item response data. ACT Research Report Series, 97-4. https://files.eric.ed.gov/fulltext/ED414297.pdf
• Embretson, S.E., & Reise, S. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates.
• Finch, H., & Habing, B. (2007). Performance of DIMTEST- and NOHARMbased statistics for testing unidimensionality. Applied Psychological Measurement, 31, 292-307. https://doi.org/10.1177/0146621606294490
• Fleisman, A.I. (1978). A method for simulating non-normal distributions. Psychometrika, 43, 521-532. https://doi.org/10.1007/BF02293811
• Froelich, A.G., & Habing, B. (2008). Conditional covariance-based subtest selection for DIMTEST. Applied Psychological Measurement, 32, 138 155. https://doi.org/10.1177/0146621607300421
• Gessaroli, M.E., & De Champlain, A.F. (1996). Using an approximate chi-square statistic to test the number of dimensions underlying the responses to a set of items. Journal of Educational Measurement, 33, 157 179. https://doi.org/10.1111/j.1745 3984.1996.tb00487.x Göçer Şahin, S. (2016). Examining parameter estimation when treating semi mixed multidimensional constructs as unidimensional [Unpublished doctoral dissertation]. Hacettepe University
• Hattie, J. (1985). Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9(8), 139 – 145. http://dx.doi.org/10.1177/014662168500900204
• Hattie, J., Krakowski, K., Rogers, H.J., & Swaminathan, H. (1996). An assessment of Stout’s index of essential dimensionality. Applied Psychological Measurement, 20, 1-14. https://doi.org/10.1177%2F014662169602000101
• Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods 6(1), 53-60. https://doi.org/10.21427/D7CF7R
• Hu, L-T., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. https://doi.org/10.1080/10705519909540118
• Kaya, K.Ö., & Kelecioğlu, H. (2016). The effect of sample size on parametric and nonparametric factor analytical methods. Educational Sciences: Theory & Practice. 16(1), 153-171. http://dx.doi.org/10.12738/estp.2016.1.0220
• Kim, H.R. (1994). New techniques for dimensionality assessment of standardized test data. [Unpublished doctoral dissertation]. University of Illinois at Urbana-Champaign, Department of Statistics.
• Ledasma, R.D., & Valero-Mora, P. (2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out Parallel Analysis. Practical Assessment, Research & Evaluation, 12 (2). https://doi.org/10.7275/wjnc-nm63
• Mroch, A.A., & Bolt, D.M. (2006). A simulation comparison of parametric and nonparametric dimensionality detection procedures. Applied Measurement in Education, 19 (1), 67-91. https://doi.org/10.1207/s15324818ame1901_4
• Nandakumar, R., & Stout, W. (1993). Refinement of Stout’s procedure for assessing latent trait unidimensionality. Journal of Educational Statistics, 18(1), 41 68. https://psycnet.apa.org/doi/10.2307/1165182
• Özbek Baştuğ, Ö.Y. (2012). Assessment of Dimensionality in Social Science Subtest. Educational Sciences: Theory & Practice. 12(1), Winter: 382-385.
• Reichenberg, R.E. (2013). A comparison of DIMTEST and generalized dimensionality discrepancy approaches to assessing dimensionality in item response theory [M.S. dissertation, Arizona State University, Arizona]. https://doi.org/10.3102%2F10769986018001041
• Reckase, M.D. (2009). Multidimensional item response theory. Springer Dordrecht Heidelberg.
• Roussos, L.A., & Özbek, O.Y. (2006). Formulation of the DETECT population parameter and evaluation of DETECT estimator bias. Journal of Educational Measurement, 43, 215-243. https://doi.org/10.1111/j.1745-3984.2006.00014.x
• Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617. https://doi.org/10.1007/BF02294821
• Stout, W., Habing, B., Douglas, J., Kim, H.R., Roussos, L., & Zhang J. (1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 19, 331-354. https://doi.org/10.1177%2F014662169602000403
• Sünbül, Ö. (2011). Çeşitli boyutluluk özelliklerine sahip yapılarda, madde parametrelerinin değişmezliğinin klasik test teorisi, tek boyutlu madde tepki kuramı ve çok boyutlu madde tepki kuramı çerçevesinde incelenmesi [Unpublished doctoral dissertation]. Mersin University.
• Sünbül, Ö., & Seo, M. (2012). Performance of test statistics for verifying unidimensionality, [Conference presentation abstract]. 2012 Annual Meeting, April 12-16, Vancouver, British Columbia, CANADA
• Svetina, D. (2011). Assessing dimensionality in complex data structures: A performance comparison of DETECT and NOHARM procedures [Unpublished doctoral dissertation]. Arizona State University
• Svetina, D., & Levy, R. (2014). A framework for dimensionality assessment for multidimensional item response models. Educational Assessment, 19(1), 35-57. https://doi.org/10.1080/10627197.2014.869450
• Tate, R. (2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27, 159-203. https://doi.org/10.1177/0146621603027003001
• Touron, J., Lizasoain, L., & Joaristi, L. (2012). Assessing the unidimensionality of the School and College Ability Test (SCAT, Spanish version) using non-parametric methods based on item response theory. High Ability Studies. 23(2), 183 202. https://doi.org/10.1080/13598139.2012.735401
• Zhang, B. (2008). Application of unidimensional item response models to tests with items sensitive to secondary dimension. The Journal of Experimental Education, 77 (2), 147-166. https://doi.org/10.3200/JEXE.77.2.147-166