Automatic item generation for non-verbal reasoning items

Year 2023, , 132 - 148, 27.12.2023

Ayfer Sayın , Sabiha Bozdağ Mark J. Gıerl

https://doi.org/10.21449/ijate.1359348

Abstract

The purpose of this study is to generate non-verbal items for a visual reasoning test using templated-based automatic item generation (AIG). The fundamental research method involved following the three stages of template-based AIG. An item from the 2016 4th-grade entrance exam of the Science and Art Center (known as BİLSEM) was chosen as the parent item. A cognitive model and an item model were developed for non-verbal reasoning. Then, the items were generated using computer algorithms. For the first item model, 112 items were generated, and for the second item model, 1728 items were produced. The items were evaluated based on subject matter experts (SMEs). The SMEs indicated that the items met the criteria of one right answer, single content and behavior, not trivial content, and homogeneous choices. Additionally, SMEs' opinions determined that the items have varying item difficulty. The results obtained demonstrate the feasibility of AIG for creating an extensive item repository consisting of non-verbal visual reasoning items.

Keywords

Automatic item generation, Non-verbal items, Visual reasoning test, BİLSEM

References

• Adji, T.B., Pribadi, F.S., Prabowo, H.E., Rosnawati, R., & Wijaya, A. (2018). Generating parallel mathematic items using automatic item generation. ICEAP 2019, 1(1), 89-93. https://doi.org/10.26499/iceap.v1i1.78
• Arendasy, M.E., & Sommer, M. (2012). Using automatic item generation to meet the increasing item demands of high-stakes educational and occupational assessment. Learning and individual differences, 22(1), 112-117. https://doi.org/10.1016/j.lindif.2011.11.005
• Atli, S. (2007). Matematiksel mantıksal yetenek ile ritimsel yetenek arasındaki ilişkiler [Relations between matematical-logical talent and rhythmic intelligence] [Unpublished master's thesis]. Gazi University.
• Balboni, G., Naglieri, J.A., & Cubelli, R. (2010). Concurrent and predictive validity of the raven progressive matrices and the Naglieri Nonverbal Ability Test. Journal of Psychoeducational Assessment, 28(3), 222 235. https://doi.org/10.1177/073428290934376
• Bildiren, A., Bıkmaz Bilgen, Ö., & Korkmaz, M. (2021). National non-verbal cognitive ability test (BNV) development study. SAGE Open, 11(3). https://doi.org/10.1177/2158244021104694
• Bilgiç, N., Taştan, A., Kurukaya, G., Kaya, K., Avanoğlu, O., ve Topal, T. (2017). Özel yetenekli bireylerin eğitimi strateji ve uygulama kılavuzu [Education of specially gifted individuals’ strategy and implementation guide]. MEB Özel Eğitim ve Rehberlik Hizmetleri Genel Müdürlüğü. https://orgm.meb.gov.tr/meb_iys_dosyalar/2013_11/25034903_zelyeteneklibireylerineitimistratejiveuygulamaklavuzu.pdf
• BİLSEM Online (2023a). Sıkça sorulan sorular: BİSEM sınav soruları yeteneğe göre değişir mi? [Frequently asked questions: Do BİLSEM exam questions vary depending on ability?]. https://www.bilsemonline.com/sss
• BİLSEM Online (2023b). BNV Zeka Testi Nedir? [What is BNV Intelligence Test?]. https://bilsemonline.com/blog/bnv-zeka-testi-nedir
• Choi, J., & Zhang, X. (2019). Computerized item modeling practices using computer adaptive formative assessment automatic item generation system: A tutorial. The Quantitative Methods for Psychology, 15(3), 214-225. https://doi.org/10.20982/tqmp.15.3.p214
• Cohen, R.J., & Swerdlik, M.E. (2015). Psychological testing and assessment. McGraw-Hill Education.
• Cooper, L.A. (1975). Mental rotation of random two-dimensional shapes. Cognitive Psychology, 7(1), 20-43. https://doi.org/10.1016/0010-0285(75)90003-1
• DeThorne, L.S. & Schaefer, B.A. (2004). A guide to child nonverbal IQ measures. American Journal of Speech-Language Psychology, 13(4), 275-290. https://doi.org/10.1044/1058-0360(2004/029)
• Embretson, S.E., & Kingston, N.M. (2018). Automatic item generation: A more efficient process for developing mathematics achievement items? Journal of Educational Measurement, 55(1), 112-131. https://doi.org/10.1111/jedm.12166
• Embretson, S., & Yang, X. (2007). 23 Automatic item generation and cognitive psychology. Handbook of statistics, 26, 747-768. https://doi.org/10.1016/S0169-7161(06)26023-1
• Falcão, F., Costa, P., & Pêgo, J.M. (2022). Feasibility assurance: a review of automatic item generation in medical assessment. Advances in Health Sciences Education, 27(2), 405-425. https://doi.org/10.1007/s10459-022-10092-z
• Gibbons, A., & Warne, R.T. (2019). First publication of subtests in the Stanford-Binet 5, WAIS-IV, WISC V, and WPPSI IV. Intelligence, 75, 9 18. https://doi.org/10.1016/j.intell.2019.02.005
• Gierl, M.J., & Haladyna, T. (2012). Automatic item generation: Theory and practice. Routledge.
• Gierl, M.J., & Lai, H. (2012). The role of item models in automatic item generation. International Journal of Testing, 12(3), 273 298. https://doi.org/10.1080/15305058.2011.635830
• Gierl, M.J., & Lai, H. (2013). Instructional topics in educational measurement (ITEMS) module: Using automated processes to generate test items. Educational Measurement: Issues and Practice, 32(3), 36-50. https://doi.org/10.1111/emip.12018
• Gierl, M.J. & Lai, H. (2016). Automatic item generation. In S. Lane, M. Raymond, & T. Haladyna (Eds.), Handbook of test development (2nd edition, pp. 410-429). Routledge.
• Gierl, M.J., Ball, M.M., Vele, V., & Lai, H. (2015). A method for generating nonverbal reasoning items using n-layer modeling. In Computer Assisted Assessment. Research into E-Assessment: 18th International Conference, CAA 2015, Zeist, The Netherlands, June 22–23, 2015. https://doi.org/10.1007/978-3-319-27704-2_2
• Gierl, M., Lai, H., & Tanygin, V. (2021). Advanced methods in automatic item generation. Routledge.
• Haladyna, T. M., Downing, S. M. & Rodriguez, M. C. (2002) A review of multiple-choice item-writing guidelines for classroom assessment, Applied Measurement in Education, 15(3), 309-333, https://doi.org/10.1207/S15324818AME1503_5
• Hausknecht, J.P., Halpert, J.A., Di Paolo, N.T., & Moriarty Gerrard, M.O. (2007). Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology, 92(2), 373–385. https://doi.org/10.1037/0021-9010.92.2.373
• Horn, J.L., & Cattell, R.B. (1966). Refinement and test of the theory of fluid and crystallized general intelligences. Journal of Educational Psychology, 57(5), 253 270. https://doi.org/10.1037/h0023816
• Karasar, N. (2022). Bilimsel araştırma yöntemleri (37. Basım). Nobel Yayıncılık.
• Kemer, B., & Çakan, M. (2020). Examining the validity of the psychological scales frequently used in guidance and research centers with respect to measurement standards of validity. Journal of Research in Education and Society, 7(1), 323 348. https://dergipark.org.tr/en/pub/etad/issue/55359/731549
• Kocagül, M., & Çoban, G.Ü. (2022). An evaluation on science teachers' scientific reasoning skills. Cumhuriyet International Journal of Education, 11(2), 361 373. https://doi.org/10.30703/cije.1017938
• Kosh, A.E., Simpson, M.A., Bickel, L., Kellogg, M., & Sanford‐Moore, E. (2019). A cost–benefit analysis of automatic item generation. Educational Measurement: Issues and Practice, 38(1), 48-53. https://doi.org/10.1111/emip.12237
• Kurnaz, A., & Ekici, S.G. (2020). BİLSEM tanılama sürecinde kullanılan zeka testlerinin psikolojik danışmanların ve BİLSEM öğretmenlerinin görüşlerine göre değerlendirilmesi [Evaluation of intelligence tests used in BİLSEM diagnostic process according to the opinions of psychological counselors and BİLSEM Teachers]. Çocuk ve Medeniyet, 5(10), 365-399. https://dergipark.org.tr/en/pub/cm/issue/59377/850922
• Kurtz, K., Gentner, D., & Gunn, V. (1999). Reasoning. In B. M. Bly & D. E. Rumelhart (Eds), Cognitive science, pp. 145-200. California: Academic Press.
• Lai, H., Gierl, M.J., Byrne, B.E., Spielman, A.I., & Waldschmidt, D.M. (2016). Three modeling applications to promote automatic item generation for examinations in dentistry. Journal of Dental Education, 80(3), 339-347. PMID: 26933110.
• Lawson, A.E. (2004). The nature and development of scientific reasoning: A synthetic view. International Journal of Science and Mathematics Education, 2(3), 307-338. https://doi.org/10.1007/s10763-004-3224-2
• Leighton, J.P. (2012). Learning sciences, cognitive models, and automatic item generation. In M.J. Gierl, & T.M. Haladyna (Eds), Automatic item generation: Theory and practice, pp. 121-135. Routledge.
• Lewis, J.D., DeCamp-Fritson, S.S., Ramage, J.C., McFarland, M.A., & Archwamety, T. (2007). Selecting for ethnically diverse children who may be gifted using Raven’s Standard Progressive Matrices and Naglieri Nonverbal Abilities Test. Multicultural Education, 15(1), 38-42.
• Lohman, D.F., & Hagen, E. (2003). Interpretive guide for teachers and counselors: cognitive abilities test Form 6-all levels. ITASCA, Illinois: Riverside Publishing.
• Mercan, Z. (2021). Studies on early childhood reasoning skills in Turkey. Journal of Muallim Rıfat Faculty of Education, 3(2), 104-120.
• Ministry of National Education (MoNE) (2015). Bilim ve Sanat Merkezleri Yönergesi [Science and Art Centers Directive]. MoNE General Directorate of Special Education and Guidance Services. https://orgm.meb.gov.tr/meb_iys_dosyalar/2015_10/26091626_blsemkilavuz26.10.2015.pdf
• Ministry of National Education (MoNE) (2021). Bilim ve Sanat Merkezleri Yönergesi [Science and Art Centers Directive]. MoNE General Directorate of Special Education and Guidance Services]. https://orgm.meb.gov.tr/meb_iys_dosyalar/2021_12/30144032_2021 2022_YILI_BILIM_VE_SANAT_MERKEZLERI_OGRENCI_TANILAMA_VE_YERLESTIRME_KILAVUZU.pdf
• Ministry of National Education (MoNE) (2022a). Bilim ve Sanat Merkezleri Yönergesi [Science and Art Centers Directive]. MoNE General Directorate of Special Education and Guidance Services. https://orgm.meb.gov.tr/meb_iys_dosyalar/2016_10/07031350_bilsem_yonergesi.pdf
• Ministry of National Education (MoNE) (2022b). Bilim ve Sanat Merkezleri öğrenci tanılama ve yerleştirme kılavuzu [Science and Art Centers student identification and placement guide]. MoNE General Directorate of Special Education and Guidance Services. https://orgm.meb.gov.tr/www/bilsem-ogrenci-tanilama-ve-yerlestirme-kilavuzu-yayimlandi/icerik/2154
• Mullin, I.V.S., Martin, M.O., & Foy, P. (2005). IEA’s TIMSS 2003 international report on achievement in the mathematics cognitive domains. TIMSS & PIRLS Internatonal study Center. Lynch School of Education, Boston College. https://files.eric.ed.gov/fulltext/ED494652.pdf
• Naglieri, J.A., & Ford, D.Y. (2005). Increasing minority children’s representation in gifted education: A response to Lohman. Gifted Child Quarterly, 49(1), 29 36. https://doi.org/10.1177/001698620504900104
• Nolte, N., Schmitz, F., Fleischer, J., Bungart, M., & Leutner, D. (2022). Rotational complexity in mental rotation around cardinal and skewed rotation axes. Intelligence, 91, 101626. https://doi.org/10.1016/j.intell.2022.101626
• Pylyshyn, Z. W. (1979). The rate of “mental rotation” of images: A test of a holistic analogue hypothesis. Memory & Cognition, 7(1), 19-28. https://doi.org/10.3758/BF03196930
• Ryoo, J.H., Park, S., Suh, H., Choi, J., & Kwon, J. (2022). Development of a new measure of cognitive ability using automatic item generation and its psychometric properties. SAGE Open, 1-13. https://doi.org/10.1177/21582440221095016
• Sak, U., Sezerel, B.B., Dulger, E., Sozel, K., & Ayas, M.B. (2019). Validity of the Anadolu-Sak Intelligence Scale in the identification of gifted students. Psychological Test and Assessment Modeling, 61(3), 263-283. https://psycnet.apa.org/record/2020-53108-001
• Sayın, A., & Gierl, M.J. (2023). Automatic item generation for online measurement and evaluation: Turkish literature items. International Journal of Assessment Tools in Education, 10(2), 218-231. https://doi.org/10.21449/ijate.1249297
• Shin, E. (2021). Automated item generation by combining the non-template and template-based approaches to generate reading inference test items. [Doctoral dissertation, University of Alberta]. Education and Research Archive. https://doi.org/10.7939/r3-75wr-hc80
• Sinharay, S., & Johnson, M. (2005). Analysis of data from an admissions test with item models. ETS Research Report Series, 2005(1), 1 32. https://files.eric.ed.gov/fulltext/EJ1111287.pdf
• Tamul, Ö.F., Sezerel, B.B., Sak, U., & Karabacak, F. (2020). Social validity study of the Anadolu-SAK intelligence scale (ASIS). PAU Journal of Education, 49, 393-412. https://doi.org/10.9779/pauefd.575479
• Weiss, L.C., Saklofske, D.H., Holdnack, J.A., & Prifitera, A. (2016). WISC-V: Advances in the assessment of intelligence. In L.G. Weiss, D.H. Saklofske, J.A. Holdnack, & A. Prifitera (Eds.), WISC-V assessment and interpretation: Scientist-practitioner perspectives (pp. 3–23). Elsevier Academic Press. https://doi.org/10.1016/B978-0-12-404697-9.00001-7