Developing A Computerized Adaptive Test Form of the Occupational Field Interest Inventory

Year 2023, Volume: 14 Issue: 1, 47 - 61, 25.03.2023

Volkan Alkan , Kaan Zulfikar Deniz

https://doi.org/10.21031/epod.1153713

Abstract

In this research, the aim was to apply the Occupational Field Interest Inventory (OFII), which was developed in paper-pencil format, as a Computerized Adaptive Test (CAT). For this purpose, the paper and pencil form of the OFII was applied to 1425 high school students and post-hoc simulations were carried out with the obtained data. According to results obtained from the simulations, it was decided that the most ideal criteria for the CAT application were GPCM as the IRT model, .40 standard error value as the test termination rule, and MFI as the item selection method. The OFII ended with an average of 59 items, and the correlations between scores obtained from the paper-pencil form and thetas (θ) estimated by simulation ranged between .91-.97. According to post-hoc simulation results, the CAT application was applied to 150 students. It was observed that the correlations between the scores of students from the online application of the paper-pencil form and θ levels estimated by the CAT form varied between .73 and .91.

Keywords

Computerized adaptive test, Item Response Theory, Occupational Field Interest Inventory, occupational interest

References

• Achtyes, E. D., Halstead, S., Smart, L., Moore, T., Frank, E., Kupfer, D. J., & Gibbons, R. D. (2015). Validation of computerized adaptive testing in an outpatient nonacademic setting: The vocations trial. Psychiatric Services, 1-6. https://doi.org/10.1176/appi.ps.201400390
• Akar, C. (2012). Factors affecting university choice: A study on students of economics and administrative sciences. Journal of Eskişehir Osmangazi University Faculty of Economics and Administrative Sciences, 7(1), 97-120.
• Altın, M. (2020). Education, status and social mobility in Turkey. Mecmua, 10, 180-196. https://doi.org/10.32579/mecmua.789249
• Aybek, E. C., & Çıkrıkçı, R. N. (2018). Applicability of self-assessment inventory as an individualized test in computer environment. Türk Psikolojik Danışma ve Rehberlik Dergisi, 8(50), 117-141.
• Babcock, B., & Weiss, D. (2012). Termination criteria in computerized adaptive tests: Do variable - length CATs provide efficient and effective measurement? Journal of Computerized Adaptive Testing, 1(1), 1-18. https://doi.org/10.7333/1212-0101001
• Betz, N. E., & Turner, B. M. (2011). Using Item Response Theory and Adaptive Testing in Online Career Assessment. Journal of Career Assessment, 19(3), 274–286. https://doi.org/10.1177/1069072710395534
• Boyd, A., Dodd, B., & Choi, S. (2010). Polytomous models in computerized adaptive testing. Nering, M., & Ostini, R. (Ed.). Handbook of polytomous item response theory models. (229-255). Routledge.
• Bulut, O., & Kan, A. (2012). Application of computerized adaptive testing to entrance examination for graduate studies in Turkey. Eurasian Journal of Educational Research, (49), 61–80.
• Chien, T.-W., Lai, W.-P., Lu, C.-W., Wang, W.-C., Chen, S.-C., Wang, H.-Y., & Su, S.-B. (2011). Web-based computer adaptive assessment of individual perceptions of job satisfaction for hospital workplace employees. BMC Medical Research Methodology, 11(1), 1-8. https://doi.org/10.1186/1471-2288-11-47
• Choi, S. W. (2009). Firestar: Computerized adaptive testing simulation program forpolytomous item response theory models. Applied Psychological Measurement, 33(8), 644–645.
• Choi, S. W., & Swartz, R. J. (2009). Comparison of CAT item selection criteria for polytomous items. Applied Psychological Measurement, 33(6), 419-440. https://doi.org/10.1177/0146621608327801
• Choi, S. W., Reise, S. P., Pilkonis, P. A., Hays, R. D., & Cella, D. (2010). Efficiency of static and computer adaptive short forms compared to full-length measures of depressive symptoms. Quality of Life Research, 19(1), 125–136. https://doi.org/10.1007/s11136-009-9560-5
• Cömert, M. (2008). Development of computer-aided assessment and evaluation software adapted to the individual. Unpublished Master's Thesis, Bahçeşehir University Institute of Science and Technology, İstanbul.
• Deniz, K. Z. (2009). Occupational Interest Inventory (OFII) development study. Yüzüncü Yıl Üniversitesi Eğitim Fakültesi Dergisi, 6(1), 289-310.
• Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates, Inc.
• Eroğlu, M. G., & Kelecioğlu, H. (2015). Comparison of Different Termination Rules in terms of Measurement Accuracy and Test Length in Individualized Computerized Testing Applications. Uludağ Üniversitesi Eğitim Fakültesi Dergisi, 28(1), 31-52.
• Gibbons, R. D., Weiss, D. J., Pilkonis, P. a, Frank, E., Moore, T., Kim, J. B., & Kupfer, D. J. (2012). Development of a computerized adaptive test for depression. Archives of General Psychiatry, 69(11), 1104-12. https://doi.org/10.1001/archgenpsychiatry.2012.14
• Gnambs, T., & Batinic, B. (2011). Polytomous adaptive classification testing: Effects of item pool size, test termination criterion, and number of cutscores. Educational and Psychological Measurement, 71(6), 1006–1022. https://doi.org/10.1177/0013164410393956
• Hambleton, R., & Swaminathan, R. (1985). Fundementals of item response theory. Sage Publications, Inc.
• Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Sage Publications, Inc.
• Harvey, R. J., & Hammer, A. L. (1999). Item response theory. The Counseling Psychologist, 27(3), 353-383.
• Hol, M. A., Vorst, H. C. ve Mellenbergh, G. J. (2007). Computerized Adaptive Testing for Polytomous Motivation Items: Administration Mode Effects and a Comparison with Short Forms. Applied Psychological Measurement, 31(5), 412–429. https://doi.org/10.1177/0146621606297314
• İşeri, A. I. (2002). Assessment of students' mathematics achievement through computer adaptive testing procedures. Unpublished Doctoral Dissertation, Middle East Technical University, Ankara.
• Jodoin, M. G., Zenisky, A., & Hambleton, R. K. (2006). Comparison of the psychometric properties of several computer-based test designs for credentialing exams with multiple purposes. Applied Measurement in Education, 19(3), 203–220. https://doi.org/10.1207/s15324818ame1903_3
• Kalender, İ. (2012). Computerized adaptive testing for student selection to higher education. Yükseköğretim Dergisi, 2(1), 13-19.
• Kezer, F, & Koç, N. (2014). Comparison of Individualized Test Strategies in Computer Environment. Eğitim Bilimleri Araştırmaları Dergisi, 4(1), 145-174. https://doi.org/10.12973/jesr.2014.41.8
• Kocalevent, R. D., Rose, M., Becker, J., Walter, O. B., Fliege, H., Bjorner, J. B., ... & Klapp, B. F. (2009). An evaluation of patient-reported outcomes found computerized adaptive testing was efficient in assessing stress perception. Journal of Clinical Epidemiology, 62(3), 278-287.
• McDonald, P. L. (2002). Computer adaptive test for measuring personality factors using item response theory. Unpublished Doctoral Dissertation. The University Western of Ontario, London.
• Meyer, J. P. (2010). Understanding measurement: Reliability. Oxford University Press.
• Ostini, R., & Nering, M. L. (Eds.). (2010). Polytomous item response theory models. Taylor and Francis Group.
• Özbaşı, D., & Demirtaşlı, N. (2015). Developing the computer literacy test as an individualized test in the computer environment. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 6(2), 218-237.
• Özkan, Y. Ö. (2014). A comparison of estimated achievement scores obtained from student achievement assessment test utilizing classical test theory, unidimensional and multidimensional IRT. International Journal of Human Sciences, 11(1), 20-44.
• Öztuna, D. (2008). Application of computer adaptive testing method in disability assessment of musculoskeletal problems. Unpublished Doctoral Dissertation. Ankara University Institute of Health Sciences, Ankara.
• Rezaie, M., & Golshan, M. (2015). Computer adaptive test (CAT): Advantages and limitations. International Journal of Educational Investigations, 2(5), 128–137.
• Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Test of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23-74.
• Scullard, M. G. (2007). Application of item response theory based computerized adaptive testing to the strong interest inventory. Unpublished Doctoral Dissertation. University of Minnesota, USA.
• Simms, L. J., & Clark, L. A. (2005). Validation of a computerized adaptive version of the Schedule for Nonadaptive and Adaptive Personality (SNAP). Psychological Assessment, 17(1), 28–43. https://doi.org/10.1037/1040-3590.17.1.28
• Smits, N., Cuijpers, P., & van Straten, A. (2011). Applying computerized adaptive testing to the CES-D scale: A simulation study. Psychiatry Research, 188(1), 147–155. https://doi.org/10.1016/j.psychres.2010.12.001
• Stochl, J., Böhnke, J. R., Pickett, K. E., & Croudace, T. J. (2016). An evaluation of computerized adaptive testing for general psychological distress: combining GHQ-12 and Affectometer-2 in an item bank for public mental health research. BMC Medical Research Methodology, 16(1), 58. https://doi.org/10.1186/s12874-016-0158-7
• Şahin, A., & Özbaşı, Ö. (2017). Effects of Content Balancing and Item Selection Method on Ability Estimation in Computerized Adaptive Tests. Eurasian Journal of Educational Research, 69, 21-36.
• Şimşek, A. S. (2017). Adaptation of skills confidence occupational interest inventory and development of computerized individualized testing. Unpublished Doctoral Dissertation, Ankara University Institute of Educational Sciences, Ankara.
• Weiss, D. J. (2011). Better data from better measurements using computerized adaptive testing. Journal of Methods and Measurement in the Social Sciences, 2(1), 1-23.
• Van der Linden, W. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63(2), 201-216.
• Veldkamp, B. P. (2003). Item selection in Polytomous CAT. In Yanai H., Okada A., Shigemasu K., Kano Y., & Meulman J. J. (Eds.), New Developments in Psychometrics (pp. 207-214). Springer Verlag.
• Yoo, J. H. (2016). The effect of professional development on teacher efficacy and teachers’ self-analysis of their efficacy change. Journal of Teacher Education for Sustainability, 18(1), 84–94. https://doi.org/10.1515/jtes-2016-0007