Test Boyutluluğunun Analizinde Kullanılan Programların İncelenmesi

Year 2015, Volume: 6 Issue: 2, 0 - 0, 02.01.2016

Güler Yavuz Nuri Doğan

https://doi.org/10.21031/epod.18173

Abstract

Son yıllarda çok boyutlu veri setlerine ve çok boyutlu veri setleriyle yapılan çeşitli test uygulamalarına (test eşitleme, alt test puanlama, değişen madde fonksiyonu gibi) olan ilgi önemli düzeydedir. Bu çalışmada test boyutluluğunun analizinde ve çok boyutlu veri setleriyle çeşitli test uygulamalarında yaygın kullanılan bazı popüler paket programları (Mplus, NOHARM, TESTFACT, IRTPRO, flexMIRT, BMIRT, DIMTEST, DETECT, CCPROX/HAC, Velicer’ in MAP testi ve Paralel Analiz) incelenmiştir. Bu programların ulaşılabilirliği, hangi tür veri setleri, modeller, kestirim teknikleri ile analiz yapabildikleri belirlenmiştir. Programların boyutluluk analizini hangi tür yaklaşımla (açımlayıcı, doğrulayıcı) yaptıkları ve program çıktılarında bulunan indekslerin nasıl yorumlandıkları incelenmiştir. Ayrıca programların çeşitli özellikleri açısından karşılaştırılmalarına yönelik yapılmış çalışmalara yer verilmiştir. 

References

• Ackerman, T. A. (1989). Unidimensional IRT calibration of compensatory and noncompensatory multidimensional items. Applied Psychological Measurement, 13, 113-127.
• Ackerman, T. A. (1996) Graphical representation of multidimensional item response theory analyses. Applied Psychological Measurement, 20, 311-329.
• Ackerman, T. A., Gierl, M. J., & Walker, C.M. (2003) Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22, 37–51.
• Adams, R. J., Wilson, M. R., & Wang, W. C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23.
• Ansley, R. A., & Forsyth, T. N. (1985). An examination of the characteristics of unidimensional IRT parameter estimates derived from two-dimensional data. Applied Psychological Measurement, 9, 37-48.
• Asparouhov, T. & Muthén, B. (2012). Using Mplus TECH11 and TECH14 to test the number of latent classes. Mplus Web Notes, 14, 22.
• Beguin, A. A., & Glas, C. A. W. (2001). MCMC estimation and some modelfit analysis of multidimensional IRT models. Psychometrika, 66, 541–562.
• Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Applications of an EM algorithm. Psychometrika 46, 443-459.
• Bock, R. D., Gibbons, R., & Muraki, E. (1988) Full information item factor analysis. Applied Psychological Measurement 12, 261-280.
• Bock, R. D., & Schilling, S. G. (2003). IRT based item factor analysis. In M du Toit (ed) IRT from SSI: BILOG-MG, MULTILOG, PARSCALE, TESTFACT, 584-591. Scientific Software International, Lincolnwood, IL.
• Bock, R. D, Gibbons, R., Schilling, S. G., Muraki, E., Wilson, D. T., & Wood, R. (2003). TESTFACT 4.0 [Computer software and manual]. Lincolnwood, IL: Scientific Software International.
• Bolt, D. M., & Lall, V. F. (2003). Estimation of compensatory and noncompensatory multidimensional item response models using Markov chain Monte Carlo. Applied Psychological Measurement, 29, 395–414.
• Bradlow, E. T., Wainer, H., & Wang, X. (1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153–168.
• Cai, L., du Toit, S. H. C., & Thissen, D. (2011). IRTPRO: Flexible, Multidimensional, Multiple Categorical IRT Modeling. Scientfic Software International.
• Cai, L. (2013). flexMIRT version 2.00: A numerical engine for flexible multilevel multidimensional item analysis and test scoring. [Computer software]. Chapel Hill, NC: Vector Psychometric Group.
• Chen, W., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265‐289.
• Chib, S. & Greenberg, E. (1995). Understanding the Metropolis Hastings Algorithm. American Statistical Journal, 49, 327–335.
• Crawford, A. V., Green, S. B., Levy, R., Lo, W. J., Scott, L., Svetina, D., & Thompson, M. S. (2010). Evaluation of parallel analysis methods for determining the number of factors. Educational and Psychological Methods, 70, 885-901.
• DeChamplain, A. F. & Gessaroli, M. E. (1991, April). Assessing test dimensionality using an index based on non-linear factor analysis. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL. (ERIC Document Reproduction Service No. 334 235).
• DeChamplain, A. & Tang, L. (1993, April). The effect of non-normal ability distributions on the assessment of dimensionality. Paper presented at the annual meeting of the National Council on Measurement in Education, Atlanta, GA.
• De Champlain, A. F. (1996). The effect of multidimensionality on IRT true-score equating for subgroups of examinees. Journal of Educational Measurement, 33, 181-201.
• Deng, H. & Ansley, T. (2000, April). Detecting compensatory and noncompensatory multidimensionality using DIMTEST. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA (ERIC Document Reproduction Service No. 445 029. Retrieved from http://catalogue.nla.gov.au/Record/5673279.
• Deng, N., Han, K., T. & Hambleton, R., K., (2013). A Reviewof DIMPACK Version 1.0: Conditional Covariance Based Test Dimensionality Analysis Package. Applied Psychological Measurement, 37 (2), 162-172.
• Drasgow, F., & Parsons, C. K. (1983). Application of unidimensional item response theory models to multidimensional data. Applied Psychological Measurement, 7, 189-199.
• du Toit, M. (2003). IRT from SSI: BILOG-MG, MULTILOG, PARSCALE, TESTFACT. Lincolnwood, IL: Scientific Software International.
• Embretson, S. E. (1997). Multicomponent response models. In W. J. Van der Linden R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 305–321). New York: Springer Verlag.
• Finch, H., & Habing, B. (2005). Comparison of NOHARM and DETECT in item cluster recovery: Counting dimensions and allocating items. Journal of Educational Measurement, 42, 149-169.
• Finch, H., & Habing, B. (2007). Performance of DIMTEST- and NOHARM-based statistics for testing unidimensionality. Applied Psychological Measurement, 31, 292-307.
• Fraser, C. (1988). NOHARM II: A Fortran program for fitting unidimensional and multidimensional normal ogive models of latent trait theory [Software]. Armidale, New South Wales: University of New England, Centre for Behavioral Studies.
• Fraser, C., & McDonald, R. P. (1988). NOHARM: Least squares item factor analysis. Multivariate Behavioral Research, 23, 267-269.
• Gamerman, D. & Lopes, H. F. (2006) Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference. Second Edition. London: Chapman & Hall/CRC Press.
• Garrido, L. E., Abad, F. J., & Ponsoda, V. (2011). Performance of Velicer’s Minimum Average Partial Factor Retention Method with Categorical Variables. Educational and Psychological Measurement, 71 (3), 551-570.
• Gelman, A., Meng, X. L. & Stern, H. S. (1996). Posterior predictive assessment of model fitness via realized discrepancies (with discussion). Statist. Sinica 6, 733–760.
• Gessaroli, M. E., & De Champlain, A. F. (1996). Using an approximate chi-square statistic to test the number of dimensions underlying the responses to a set of items. Journal of Educational Measurement, 33, 157-192.
• Gosz, J. & Walker, C. M. (2001). An Empirical Comparison of Multidimensional Item Response Data Using TESTFACT and NOHARM. Annual meeting of the American Educational Research Association, New Orleans, LA, USA.
• Hattie, J., Krakowski, K., Rogers, J., & Swaminathan, H. (1996). An assessment of Stout's index of essential dimensionality. Applied Psychological Measurement, 20, 1-14.
• Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7 (2), 191-205.
• Harrison, D. A. (1986). Robustness of IRT parameter estimation to violations on the unidimensionality assumption. Journal of Educational Statistics, 11, 91–115.
• Horn, J. L. (1965). A rationale and test for rhe number of factors in factor analysis. Psychometrika, 30 (2). 179-185.
• Houts, C. R., & Cai, L. (2013). flexMIRT user’s manual version 2.0: flexible multilevel multidimensional item analysis and test scoring. Chapel Hill, NC: Vector Psychometric Group.
• Jang, E. E., & Roussos, L. A. (2007). An investigation into the dimensionality of TOEFL using conditional covariance-based non-parametric approach. Journal of Educational Measurement, 44 (1), 1-21.
• Justicia, F., Pichardo, M. C., Cano, F., Berben, A. B. G., & De la Fuente, J. (2008). The revised two- factor study process questionnaire (R-SPQ-2F). Exploratory and confimatory factor analyses at item level. Europen Journal of Psychological of Education, 23 (3), 355-372.
• Kim, H.R. (1994). New techniques for the dimensionality assessment of standardized test data. (Doctoral Dissertation, University of Illinois at Urbana—Champaign).
• Kline, R. B. (2005). Principles and Practice of Structural Equation Modeling (2nd ed.). New York: Guilford. 366 pp., ISBN 978-1-57230-690-5.
• Knol, D. L. & Berger, M. P. (1991). Empirical comparison between factor analysis and multidimensional item response models. Multivariate Behavioral Research. 26 (3), 457-477.
• Ladesma, R.D., & Valero- Mora, P. (2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out parallel analysis. Practical Assessment. Research and Evaluation, 12, 1-11.
• McDonald, R. P. (1962). A general approach to nonlinear factor analysis. Psychometrika, 27, 398-415.
• McDonald, R. P. (1967). Nonlinear factor analysis [Psychometric Monographs, No. 15]. Chicago: University of Chicago Press.
• McDonald, R. P. (1981). The dimensionality of tests and items. British Journal of Mathematical and Statistical Psychology, 34, 100-117.
• McDonald, R. P. (1997). Normal-ogive multidimensional model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 257-269). New York, NY: Springer-Verlag.
• McDonald, R. P. (2000). A basis for multidimensional item response theory. Applied Psychological Measurement, 24, 99-114.
• Meara, K., Robin, F., & Sireci, S.G. (2000). Using Multidimensional Scaling to Assess the Dimensionality of Dichotomous Item Data, Multivariate Behavioral Research, 35 (2), 229–259.
• Munroe, A., & Pearson, C. (2006). The Munroe Multicultural Attitude Scale Questionnaire: A new instrument for multicultural studies. Educational and Psychological Measurement, 66, 819-834.
• Muthén, L.K. & Muthén, B.O. (1998-2012). Mplus User’s Guide. Seventh Edition. Los Angeles, CA: Muthén & Muthén.
• Nelson, J. M., Canivez, G. L., Lindstrom, W., & Hatt, C. V. (2007). Higher-order exploratory factor analysis of the Reynolds Intellectual Assessment Scales with a referred sample. Journal of School Psychology, 45, 439–456.
• Nandakumar, R. & Stout, W. (1993). Refinements of stout's procedure for assessing latent trait unidimensionality. Journal of Educational Statistics, 18 (1), 41-68.
• Nonparametric Dimensionality Assessment Package (2006)., Champaign, IL: William Stout Institute for Measurement.
• O’Connor, B.P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instruments, & Computers, 32, 396-402.
• Özer Özkan, Y., & Acar Güvendir M. (2014). Türkiye’de Uygulanan Geniş Ölçekli Testlerin Çok Boyutluluğunun Analizi. Mehmet Akif Ersoy Üniversitesi Eğitim Fakültesi Dergisi, 29, 31-47.
• Piccone, A.V. (2009). A comparison Of Three Computational Procedures for Solving the Number of factors problem in exploratory factor analysis. (Doctoral dissertation. University of Northern Colorado).
• Reckase, M. D. (1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 27, 25-36.
• Reckase, M. D. (2009). Multidimensional Item Response Theory. Springer-Verlag, New York.
• Roussos, L. A. (1992). Hierarchical agglomerative clustering computer program user’s manual. (Doctoral Dissertation, University of Illinois, Urbana-Champaign).
• Roussos, L. A., Stout, W. F., & Marden, J. I. (1998). Using new proximity measures with hierarchical cluster analysis to detect multidimensionality. Journal of Educational Measurement, 35, 1-30.
• Roussos, L.A. & Ozbek, O. (2006). Formulation of the DETECT population parameter and evaluation of DETECT estimator bias. Journal of Educational Measurement, 43, 215-243.
• Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the Reliability of Testlet‐Based Tests. Journal of Educational Measurement, 28 (3), 237-247.
• Shealy, R., & Stout,W. (1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DIF as well as item bias/DIF. Psychometrika, 58, 159-194.
• Stone, C. A., & Yeh, C. C. (2006). Assessing the dimensionality and factor structure of multiple-choice exams: An empirical comparison of methods using the Multistate Bar Examination. Educationa and Psychological Measurement, 66, 193–214.
• Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617.
• Stout, W., Froelich, A. G., & Gao, F. (2001). Using resampling methods to produce an improved DIMTEST procedure. In A. Boomsma, M. A. J. van Duijn, & T. A. B. Snijders (Eds.), Essays on item response theory (pp. 357-375). New York, NY: Springer-Verlag.
• Stout, W. (2006). DIMPACK 1.0. Chicago, IL: Assessment Systems Corporation.
• Storch, E. A., Murphy, T. K., Bagner, D. M., Johns, N., Baumeister, A., & Goodman, W. K. (2006). Reliability and Validity of the child behavior checklist obsessive-compulsive scale. Psychistry Research, 129, 91-98.
• Svetina, D. (2011). Assessing Dimensionality in Complex Data Structures: A Performance Comparison of DETECT and NOHARM Procedures. (Doctoral dissertation, Arizona State University). Retrieved from http://repository.asu.edu/attachments/56763/content.
• Svetina, D., & Levy, R. (2012). An Overview of Software for Conducting Dimensionality Assessment in Multidimensional Models. Applied Psychological Measurement, 36 (8), 659-669.
• Tate, R. (2003). A comparison of selected empirical methods for assessing the structure of responses to test items. Applied Psychological Measurement, 27, 159-203.
• Velicer, W.F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrka, 41, 321-327.
• Velicer, W. F., Eaton, C.A., & Fava, J.L. (2000). Construct Explication through Factor or Component Analysis: A Review and Evaluation of Alternative Procedures for Determining the Number of Factors or Components. In Goffin, R. D., & Helmes, E. (Eds.), Problems and Solutions in Human Assessment: Honoring Douglas Jackson at Seventy. Boston: Kluwer. (pp. 41-71).
• Walker, C. M., & Beretvas, S. N. (2003). Comparing multidimensional and unidimensional proficiency classifications: Multidimensional IRT as a diagnostic aid. Journal of Educational Measurement, 40, 255-275.
• Watkins, M. W. (2006). Determining parallel analysis criteria. Journal of Modern Applied Statistical Methods, 5, 344-346.
• Way, W. D., Ansley, T. N., & Forsyth, R. A. (1986). The effects of two-dimensional data on unidimensional IRT parameter estimates. Annual meeting of the American Educational Research Association, San Francisco CA
• Yavuz, G. (2014). Çok Boyutlu Madde Tepki Kuramı Modelleri ve Programları için Karşılaştırmalı Analizler. (Yayımlanmamış Doktora tezi, Hacettepe Üniversi¬tesi, Eğitimde Ölçme ve Değerlendirme Anabilim Dalı, Ankara).
• Yen, W. M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30, 187–213.
• Yao, L. (2003). BMIRT: Bayesian multivariate item response theory. CTB/McGraw-Hill, Monterey, CA.
• Yao, L., & Boughton, K. A. (2009). Multidimensional Linking for Tests with Mixed Item Types. Journal of Educational Measurement, 46 (2), 177-197.
• Yao, L. (2013). The BMIRT Toolkit. Defense Manpower Data Center, DoD Center Monterey Bay,US.
• Yeh, C. (2007). The effect of guessing on assessing dimensionality in multiple-choice tests: A Monte Carlo study with application. (Doctoral Dissertation. University of Pittsburg). Retrieved from http://d-scholarship.pitt.edu/7286/.
• Zhang, J., & Stout, W. (1999).The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213-249.
• Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the numberof components to retain. Psychological Bulletin, 99, 432-442.