The purpose of this study was to investigate the Type I Error findings and power rates of the methods used to determine dimensionality in unidimensional and bidimensional psychological constructs for various conditions (characteristic of the distribution, sample size, length of the test, and interdimensional correlation) and to examine the joint effect of the conditions (effect of the interaction of conditions) as well as the main effect of each condition. The simulative data were generated for the study using the SAS program. Within the scope of the study, the data were analyzed using the DIMTEST T statistic and the Dimensionality DETECT IDN index, which is one of the non-parametric methods. The Nonlinear Factor Analysis (NOHARM) method was preferred from among parametric methods. As a result of the study, it was noted that the most consistent results in making the unidimensionality decisions belong to the Nonlinear Factor Analysis method showing standard normal distribution according to the shape of the distribution. When the power study results were examined, it was noted that the DIMTEST T statistic gave more accurate results in conditions with large samples, consisting of data with standard normal distribution. On the other hand, while results of the DETECT IDN index and Nonlinear factor analysis were more internally consistent, it was noted that in conditions where the sample size was 1000 and above, the DIMTEST T statistic also made the right decisions in determining dimensionality.
Ackerman, T.A. (1994). Using multidimensional item response theory to understand what items and tests are measuring. Applied Measurement in Education, 7(4), 255-278. https://doi.org/10.1207/s15324818ame0704_1
Ackerman, T.A., Gierl, M.J., & Walker, C.M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22(3), 37-51. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
Davey, T., Nering M.L., & Thompson, T. (1997). Realistic simulation of item response data. ACT Research Report Series, 97-4. https://files.eric.ed.gov/fulltext/ED414297.pdf
Embretson, S.E., & Reise, S. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates.
Finch, H., & Habing, B. (2007). Performance of DIMTEST- and NOHARMbased statistics for testing unidimensionality. Applied Psychological Measurement, 31, 292-307. https://doi.org/10.1177/0146621606294490
Fleisman, A.I. (1978). A method for simulating non-normal distributions. Psychometrika, 43, 521-532. https://doi.org/10.1007/BF02293811
Froelich, A.G., & Habing, B. (2008). Conditional covariance-based subtest selection for DIMTEST. Applied Psychological Measurement, 32, 138 155. https://doi.org/10.1177/0146621607300421
Gessaroli, M.E., & De Champlain, A.F. (1996). Using an approximate chi-square statistic to test the number of dimensions underlying the responses to a set of items. Journal of Educational Measurement, 33, 157 179. https://doi.org/10.1111/j.1745 3984.1996.tb00487.x
Göçer Şahin, S. (2016). Examining parameter estimation when treating semi mixed multidimensional constructs as unidimensional [Unpublished doctoral dissertation]. Hacettepe University
Hattie, J. (1985). Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9(8), 139 – 145. http://dx.doi.org/10.1177/014662168500900204
Hattie, J., Krakowski, K., Rogers, H.J., & Swaminathan, H. (1996). An assessment of Stout’s index of essential dimensionality. Applied Psychological Measurement, 20, 1-14. https://doi.org/10.1177%2F014662169602000101
Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods 6(1), 53-60. https://doi.org/10.21427/D7CF7R
Hu, L-T., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. https://doi.org/10.1080/10705519909540118
Kaya, K.Ö., & Kelecioğlu, H. (2016). The effect of sample size on parametric and nonparametric factor analytical methods. Educational Sciences: Theory & Practice. 16(1), 153-171. http://dx.doi.org/10.12738/estp.2016.1.0220
Kim, H.R. (1994). New techniques for dimensionality assessment of standardized test data. [Unpublished doctoral dissertation]. University of Illinois at Urbana-Champaign, Department of Statistics.
Ledasma, R.D., & Valero-Mora, P. (2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out Parallel Analysis. Practical Assessment, Research & Evaluation, 12 (2). https://doi.org/10.7275/wjnc-nm63
Mroch, A.A., & Bolt, D.M. (2006). A simulation comparison of parametric and nonparametric dimensionality detection procedures. Applied Measurement in Education, 19 (1), 67-91. https://doi.org/10.1207/s15324818ame1901_4
Nandakumar, R., & Stout, W. (1993). Refinement of Stout’s procedure for assessing latent trait unidimensionality. Journal of Educational Statistics, 18(1), 41 68. https://psycnet.apa.org/doi/10.2307/1165182
Özbek Baştuğ, Ö.Y. (2012). Assessment of Dimensionality in Social Science Subtest. Educational Sciences: Theory & Practice. 12(1), Winter: 382-385.
Reichenberg, R.E. (2013). A comparison of DIMTEST and generalized dimensionality discrepancy approaches to assessing dimensionality in item response theory [M.S. dissertation, Arizona State University, Arizona]. https://doi.org/10.3102%2F10769986018001041
Roussos, L.A., & Özbek, O.Y. (2006). Formulation of the DETECT population parameter and evaluation of DETECT estimator bias. Journal of Educational Measurement, 43, 215-243. https://doi.org/10.1111/j.1745-3984.2006.00014.x
Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617. https://doi.org/10.1007/BF02294821
Stout, W., Habing, B., Douglas, J., Kim, H.R., Roussos, L., & Zhang J. (1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 19, 331-354. https://doi.org/10.1177%2F014662169602000403
Sünbül, Ö. (2011). Çeşitli boyutluluk özelliklerine sahip yapılarda, madde parametrelerinin değişmezliğinin klasik test teorisi, tek boyutlu madde tepki kuramı ve çok boyutlu madde tepki kuramı çerçevesinde incelenmesi [Unpublished doctoral dissertation]. Mersin University.
Sünbül, Ö., & Seo, M. (2012). Performance of test statistics for verifying unidimensionality, [Conference presentation abstract]. 2012 Annual Meeting, April 12-16, Vancouver, British Columbia, CANADA
Svetina, D. (2011). Assessing dimensionality in complex data structures: A performance comparison of DETECT and NOHARM procedures [Unpublished doctoral dissertation]. Arizona State University
Svetina, D., & Levy, R. (2014). A framework for dimensionality assessment for multidimensional item response models. Educational Assessment, 19(1), 35-57. https://doi.org/10.1080/10627197.2014.869450
Tate, R. (2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27, 159-203. https://doi.org/10.1177/0146621603027003001
Touron, J., Lizasoain, L., & Joaristi, L. (2012). Assessing the unidimensionality of the School and College Ability Test (SCAT, Spanish version) using non-parametric methods based on item response theory. High Ability Studies. 23(2), 183 202. https://doi.org/10.1080/13598139.2012.735401
Zhang, B. (2008). Application of unidimensional item response models to tests with items sensitive to secondary dimension. The Journal of Experimental Education, 77 (2), 147-166. https://doi.org/10.3200/JEXE.77.2.147-166
A Comparison of type I error and power rates in procedures used determining test dimensionality
The purpose of this study was to investigate the Type I Error findings and power rates of the methods used to determine dimensionality in unidimensional and bidimensional psychological constructs for various conditions (characteristic of the distribution, sample size, length of the test, and interdimensional correlation) and to examine the joint effect of the conditions (effect of the interaction of conditions) as well as the main effect of each condition. The simulative data were generated for the study using the SAS program. Within the scope of the study, the data were analyzed using the DIMTEST T statistic and the Dimensionality DETECT IDN index, which is one of the non-parametric methods. The Nonlinear Factor Analysis (NOHARM) method was preferred from among parametric methods. As a result of the study, it was noted that the most consistent results in making the unidimensionality decisions belong to the Nonlinear Factor Analysis method showing standard normal distribution according to the shape of the distribution. When the power study results were examined, it was noted that the DIMTEST T statistic gave more accurate results in conditions with large samples, consisting of data with standard normal distribution. On the other hand, while results of the DETECT IDN index and Nonlinear factor analysis were more internally consistent, it was noted that in conditions where the sample size was 1000 and above, the DIMTEST T statistic also made the right decisions in determining dimensionality.
Ackerman, T.A. (1994). Using multidimensional item response theory to understand what items and tests are measuring. Applied Measurement in Education, 7(4), 255-278. https://doi.org/10.1207/s15324818ame0704_1
Ackerman, T.A., Gierl, M.J., & Walker, C.M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22(3), 37-51. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
Davey, T., Nering M.L., & Thompson, T. (1997). Realistic simulation of item response data. ACT Research Report Series, 97-4. https://files.eric.ed.gov/fulltext/ED414297.pdf
Embretson, S.E., & Reise, S. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates.
Finch, H., & Habing, B. (2007). Performance of DIMTEST- and NOHARMbased statistics for testing unidimensionality. Applied Psychological Measurement, 31, 292-307. https://doi.org/10.1177/0146621606294490
Fleisman, A.I. (1978). A method for simulating non-normal distributions. Psychometrika, 43, 521-532. https://doi.org/10.1007/BF02293811
Froelich, A.G., & Habing, B. (2008). Conditional covariance-based subtest selection for DIMTEST. Applied Psychological Measurement, 32, 138 155. https://doi.org/10.1177/0146621607300421
Gessaroli, M.E., & De Champlain, A.F. (1996). Using an approximate chi-square statistic to test the number of dimensions underlying the responses to a set of items. Journal of Educational Measurement, 33, 157 179. https://doi.org/10.1111/j.1745 3984.1996.tb00487.x
Göçer Şahin, S. (2016). Examining parameter estimation when treating semi mixed multidimensional constructs as unidimensional [Unpublished doctoral dissertation]. Hacettepe University
Hattie, J. (1985). Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9(8), 139 – 145. http://dx.doi.org/10.1177/014662168500900204
Hattie, J., Krakowski, K., Rogers, H.J., & Swaminathan, H. (1996). An assessment of Stout’s index of essential dimensionality. Applied Psychological Measurement, 20, 1-14. https://doi.org/10.1177%2F014662169602000101
Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods 6(1), 53-60. https://doi.org/10.21427/D7CF7R
Hu, L-T., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. https://doi.org/10.1080/10705519909540118
Kaya, K.Ö., & Kelecioğlu, H. (2016). The effect of sample size on parametric and nonparametric factor analytical methods. Educational Sciences: Theory & Practice. 16(1), 153-171. http://dx.doi.org/10.12738/estp.2016.1.0220
Kim, H.R. (1994). New techniques for dimensionality assessment of standardized test data. [Unpublished doctoral dissertation]. University of Illinois at Urbana-Champaign, Department of Statistics.
Ledasma, R.D., & Valero-Mora, P. (2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out Parallel Analysis. Practical Assessment, Research & Evaluation, 12 (2). https://doi.org/10.7275/wjnc-nm63
Mroch, A.A., & Bolt, D.M. (2006). A simulation comparison of parametric and nonparametric dimensionality detection procedures. Applied Measurement in Education, 19 (1), 67-91. https://doi.org/10.1207/s15324818ame1901_4
Nandakumar, R., & Stout, W. (1993). Refinement of Stout’s procedure for assessing latent trait unidimensionality. Journal of Educational Statistics, 18(1), 41 68. https://psycnet.apa.org/doi/10.2307/1165182
Özbek Baştuğ, Ö.Y. (2012). Assessment of Dimensionality in Social Science Subtest. Educational Sciences: Theory & Practice. 12(1), Winter: 382-385.
Reichenberg, R.E. (2013). A comparison of DIMTEST and generalized dimensionality discrepancy approaches to assessing dimensionality in item response theory [M.S. dissertation, Arizona State University, Arizona]. https://doi.org/10.3102%2F10769986018001041
Roussos, L.A., & Özbek, O.Y. (2006). Formulation of the DETECT population parameter and evaluation of DETECT estimator bias. Journal of Educational Measurement, 43, 215-243. https://doi.org/10.1111/j.1745-3984.2006.00014.x
Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617. https://doi.org/10.1007/BF02294821
Stout, W., Habing, B., Douglas, J., Kim, H.R., Roussos, L., & Zhang J. (1996). Conditional covariance-based nonparametric multidimensionality assessment. Applied Psychological Measurement, 19, 331-354. https://doi.org/10.1177%2F014662169602000403
Sünbül, Ö. (2011). Çeşitli boyutluluk özelliklerine sahip yapılarda, madde parametrelerinin değişmezliğinin klasik test teorisi, tek boyutlu madde tepki kuramı ve çok boyutlu madde tepki kuramı çerçevesinde incelenmesi [Unpublished doctoral dissertation]. Mersin University.
Sünbül, Ö., & Seo, M. (2012). Performance of test statistics for verifying unidimensionality, [Conference presentation abstract]. 2012 Annual Meeting, April 12-16, Vancouver, British Columbia, CANADA
Svetina, D. (2011). Assessing dimensionality in complex data structures: A performance comparison of DETECT and NOHARM procedures [Unpublished doctoral dissertation]. Arizona State University
Svetina, D., & Levy, R. (2014). A framework for dimensionality assessment for multidimensional item response models. Educational Assessment, 19(1), 35-57. https://doi.org/10.1080/10627197.2014.869450
Tate, R. (2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27, 159-203. https://doi.org/10.1177/0146621603027003001
Touron, J., Lizasoain, L., & Joaristi, L. (2012). Assessing the unidimensionality of the School and College Ability Test (SCAT, Spanish version) using non-parametric methods based on item response theory. High Ability Studies. 23(2), 183 202. https://doi.org/10.1080/13598139.2012.735401
Zhang, B. (2008). Application of unidimensional item response models to tests with items sensitive to secondary dimension. The Journal of Experimental Education, 77 (2), 147-166. https://doi.org/10.3200/JEXE.77.2.147-166
Güler, G., & Çıkrıkçı, N. (2022). A Comparison of type I error and power rates in procedures used determining test dimensionality. International Journal of Assessment Tools in Education, 9(3), 697-712. https://doi.org/10.21449/ijate.1059628