Using ACER ConQuest program to examine multidimensional and many-facet models

Öz

The main aim of this study was to introduce the ConQuest program, which is used in the analysis of multivariate and multidimensional data structures, and to show its applications on example data structures. To achieve this goal, a basic research approach was applied. Thus, how to use the ConQuest program and how to prepare the data set for analysis were explained step by step. Then, two example applications were made considering the multidimensional structures. Finally, different sources of variability (e.g., item, student, rater, gender), which are both multidimensional and independent of each other, were performed by considering different sources of variability together. According to the analyses, the dimensionality of the data structures must be examined in the analysis process. If the data structure is multidimensional, appropriate multidimensional IRT analyses should be performed.

Anahtar Kelimeler

• Multidimensionality
• ConQuest
• Many-facet Rasch
• Validity

References

1. Ackerman, T.A. (1994). Using multidimensional item response theory to understand what items and tests are measuring. Applied Measurement in Education, 7(4), 255–278. https://doi.org/10.1207/s15324818ame0704_1
2. Adams, R.J., Wilson, M.R., & Wang, W. (1997). The Multidimensional Random Coefficients Multinomial Logit Model. Applied Psychological Measurement, 21, 1–24. https://doi.org/10.1177%2F0146621697211001
3. Adams, R.J., Wilson, M.R., & Wu, M.L. (1997). Multilevel Item Response Models: An Approach to Errors in Variables Regression. Journal of Educational and Behavioural Statistics, 22, 46–75. https://doi.org/10.2307/1165238
4. Adams, R., Cloney, D., Wu, M., Osses, A., Schwantner, V., & Vista, A. (2022). ACER ConQuest Manual. https://conquestmanual.acer.org/
5. Adams, R.J, Wu, M.L, Cloney, D., and Wilson, M.R. (2020). ACER ConQuest: Generalised Item Response Modelling Software [Computer software]. Version 5. Camberwell, Victoria: Australian Council for Educational Research.
6. Andrich, D. (1978). A Rating Formulation for Ordered Response Categories. Psychometrika, 43, 561–573. https://doi.org/10.1007/BF02293814
7. Bartolomé, J., & Garaizar, P. (2022). Design and Validation of a Novel Tool to Assess Citizens’ Netiquette and Information and Data Literacy Using Interactive Simulations. Sustainability, 14(6), 3392. https://doi.org/10.3390/su14063392
8. Bock, D.R., & Aitkin, M. (1981). Marginal Maximum Likelihood Estimation of Item Parameters: An Application of the EM Algorithm. Psychometrika, 46, 443–459. https://doi.org/10.1007/BF02293801

9. Brnic, M., & Greefrath, G. (2021, September 13–16). Does The Gender Matter? The Use of A Dıgıtal Textbook Compared To Prınted Materıals. 15th International Conference on Technology In Mathematics Teaching (ICTMT 15), Copenhagen, Denmark.
10. Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50(2), 123-140. https://doi.org/10.1111/j.1745-3984.2012.00185.x
11. De Ayala, R.J. (2009). The theory and practice of ıtem response theory. Methodology in the Social Sciences. New York: Guildford.
12. Finch, H., & Habing, B. (2003, April). Comparison of NOHARM and DETECT in item cluster recovery: Counting dimensions and allocating items. Paper presented at the annual meeting of the National Council on Measurement, Chicago.
13. Fischer, G.H. (1983). Logistic Latent Trait Models with Linear Constraints. Psychometrika, 48, 3–26. https://doi.org/10.1007/BF02314674
14. Hahn, I. & Kähler, J. (2022). NEPS Technical Report for Science: Scaling Results of Starting Cohort 3 for Grade 11 (NEPS Survey Paper No. 93). Leibniz Institute for Educational Trajectories, National Educational Panel Study. https://doi.org/10.5157/NEPS:SP93:1.0
15. Jang, E.E., & Roussos, L.A. (2007). An investigation into the dimensionality of TOEFL using conditional covariance-based non-parametric approach. Journal of Educational Measurement, 44(1), 1-21. https://doi.org/10.1111/j.1745-3984.2007.00024.x
16. Jolin, J., & Wilson, M. (2022). Developing a Theory of Two Latent Soft Skills Progress Variables using the BEAR Assessment System: Validity Evidence for the Internal Structure of the Social Evaluative in the Workplace Instrument. Journal of Psychoeducational Assessment, 40(3), 381 399. https://doi.org/10.1177/07342829211057641
17. Jüttler, M., & Schumann, S. (2022). The long-term effects of students’ economic competencies on the transition from school to university in the international context. Research in Comparative and International Education, 17(2), 196 224. https://doi.org/10.1177/17454999221086191
18. Krell, M., Khan, S., Vergara, C., Cofré, H., Mathesius, S., & Krüger, D. (2022). Pre-Service Science Teachers’ Scientific Reasoning Competencies: Analysing the Impact of Contributing Factors. Research in Science Education, 1 21. https://doi.org/10.1007/s11165-022-10045-x
19. Koch, A., Wißhak, S., Spener, C., Naumann, A., & Hochholdinger, S. (2022). Transfer knowledge of trainers in continuing vocational education and training: Construction and piloting of a test instrument. Journal for Research on Adult Education, 1-17. https://doi.org/10.1007/s40955-022-00210-0
20. Köse, İ.A. (2012). Çok boyutlu madde tepki kuramı [Multidimensional Item Response Theory]. Journal of Measurement and Evaluation in Education and Psychology, 3(1), 221-229.
21. Linacre, J.M. (1994). Many-Facet Rasch Measurement. MESA Press.
22. Lou, J., Chen, H., & Li, R. (2022). Emotional Intelligence Scale for Male Nursing Students and Its Latent Regression on Gender and Background Variables. Healthcare, 10(5), 814. https://doi.org/10.3390/healthcare10050814
23. Masters, G.N. (1982). A Rasch Model for Partial Credit Scoring. Psychometrika, 47, 149–174. https://doi.org/10.1007/BF02296272
24. Mendoza, N.B., Cheng, E.C., & Yan, Z. (2022). Assessing teachers’ collaborative lesson planning practices: Instrument development and validation using the SECI knowledge-creation model. Studies in Educational Evaluation, 73, 101139. https://doi.org/10.1016/j.stueduc.2022.101139
25. Messick, S. (1995). Validity of psychological assessment. American Psychologist, 50(9), 741-749. https://doi.org/10.1037/0003-066X.50.9.741
26. Mischo, C., Wolstein, K., & Peters, S. (2022). Professional vision of early childhood teachers: relations to knowledge, work experience and teacher child-interaction. Early Years, 1-17. https://doi.org/10.1080/09575146.2022.2028129
27. Mroch, A.A., & Bolt, D.M. (2006). A simulation comparison of parametric and nonparametric dimensionality detection procedures. Applied Measurement in Education, 19(1), 67-91. https://doi.org/10.1207/s15324818ame1901_4
28. Oko, J. (2022). Creating a motivation scale for secondary school students in Papua New Guinea. Journal of Applied Learning and Teaching, 5(1), 1 10. https://doi.org/10.37074/jalt.2022.5.1.4
29. Osterhaus, C., Kristen-Antonow, S., Kloo, D., & Sodian, B. (2022). Advanced scaling and modeling of children’s theory of mind competencies: Longitudinal findings in 4-to 6-year-olds. International Journal of Behavioral Development, 46(3), 251-259. https://doi.org/10.1177/01650254221077334
30. Özbek-Baştuğ, O.Y. (2012). Assessment of dimensionality in social science subtest. Educational Sciences: Theory and Practice, 12(1), 375-385.
31. Özer-Özkan, Y. (2012). Öğrenci başarılarının belirlenmesi sınavından (ÖBBS) klasik test kuramı, tek boyutlu ve çok boyutlu madde tepki kuramı modelleri ile kestirilen başarı puanlarının karşılaştırılması. [A Comparison of Estimated Achivement Scores Obtained From Student Achievement Assessment Test Utilizing Classical Test Theory, Unidimensional And Multidimensional Item Response Theory Models]. [Doctoral dissertation, Ankara University]. National Thesis Center of Higher Education Board. https://tez.yok.gov.tr/UlusalTezMerkezi/
32. Özer-Özkan, Y., & Acar-Güvendir, M. (2014). The analysis of large-scale tests applied in Turkey in terms of their multidimensionality. Mehmet Akif Ersoy University Journal of Education Faculty, 1(29), 31-47.
33. Patz, R.J., & Junker, B.W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146–178. https://doi.org/10.2307/1165199
34. Rasch, G. (1980). Probabilistic Models for Some Intelligence and Attainment Test. University of Chicago Press.
35. Spink, J., Cloney, D., & Berry, A. (2022, January 01). Beyond letters and numbers: the COVID-19 pandemic and foundational literacy and numeracy in Indonesia. International Education Research. https://research.acer.edu.au/int_research/7
36. Stout, W., Froelich, A.G., & Gao, F. (2001). Using resampling methods to produce an improved DIMTEST procedure. In A. Boomsma, M.A.J. van Duijn, & T.A.B. Snijders (Eds.), Essay on item response theory (pp. 357-375). Springer. https://doi.org/10.1007/978-1-4613-0169-1_19
37. Unfried, A., Rachmatullah, A., Alexander, A., & Wiebe, E. (2022). An alternative to STEBI-A: validation of the T-STEM science scale. International Journal of STEM Education, 9(1), 1-14. https://doi.org/10.1186/s40594-022-00339-x
38. Volodin, N., & Adams, R. J. (1995). Identifying and estimating a d-dimensional item response model. International Objective Measurement Workshop, University of California.
39. Wall, S.P., Castillo, P., Shuchat-Shaw, F., Norman, E., Brown, D., Martinez-López, N., & Ravenell, J. E. (2022). Targeting versus Tailoring Educational Videos for Encouraging Deceased Organ Donor Registration in Black-Owned Barbershops. Journal of Health Communication, 27(1), 37-48. https://doi.org/10.1080/10810730.2022.2035021
40. Wang, W. (1995). Implementation and application of the multidimensional random coefficients multinomial logit. [Unpublished Doctoral dissertation]. University of California.
41. Wang, X., Yan, Z., Huang, Y., Tang, A., & Chen, J. (2022). Re-Developing the Adversity Response Profile for Chinese University Students. International Journal of Environmental Research and Public Health, 19, 6389. https://doi.org/10.3390/ijerph19116389.
42. Wilson, M.R. (1992). The ordered partition model: an extension of the partial credit model. Applied Psychological Measurement, 16, 309 325. https://doi.org/10.1177/014662169201600401
43. Wright, B.D., & Stone, M.H. (1979). Best test design: Rasch measurement. MESA Press.