Automatic item generation for non-verbal reasoning items

Abstract

The purpose of this study is to generate non-verbal items for a visual reasoning test using templated-based automatic item generation (AIG). The fundamental research method involved following the three stages of template-based AIG. An item from the 2016 4th-grade entrance exam of the Science and Art Center (known as BİLSEM) was chosen as the parent item. A cognitive model and an item model were developed for non-verbal reasoning. Then, the items were generated using computer algorithms. For the first item model, 112 items were generated, and for the second item model, 1728 items were produced. The items were evaluated based on subject matter experts (SMEs). The SMEs indicated that the items met the criteria of one right answer, single content and behavior, not trivial content, and homogeneous choices. Additionally, SMEs' opinions determined that the items have varying item difficulty. The results obtained demonstrate the feasibility of AIG for creating an extensive item repository consisting of non-verbal visual reasoning items.

Keywords

• Automatic item generation
• Non-verbal items
• Visual reasoning test
• BİLSEM

References

1. Adji, T.B., Pribadi, F.S., Prabowo, H.E., Rosnawati, R., & Wijaya, A. (2018). Generating parallel mathematic items using automatic item generation. ICEAP 2019, 1(1), 89-93. https://doi.org/10.26499/iceap.v1i1.78
2. Arendasy, M.E., & Sommer, M. (2012). Using automatic item generation to meet the increasing item demands of high-stakes educational and occupational assessment. Learning and individual differences, 22(1), 112-117. https://doi.org/10.1016/j.lindif.2011.11.005
3. Atli, S. (2007). Matematiksel mantıksal yetenek ile ritimsel yetenek arasındaki ilişkiler [Relations between matematical-logical talent and rhythmic intelligence] [Unpublished master's thesis]. Gazi University.
4. Balboni, G., Naglieri, J.A., & Cubelli, R. (2010). Concurrent and predictive validity of the raven progressive matrices and the Naglieri Nonverbal Ability Test. Journal of Psychoeducational Assessment, 28(3), 222 235. https://doi.org/10.1177/073428290934376
5. Bildiren, A., Bıkmaz Bilgen, Ö., & Korkmaz, M. (2021). National non-verbal cognitive ability test (BNV) development study. SAGE Open, 11(3). https://doi.org/10.1177/2158244021104694
6. Bilgiç, N., Taştan, A., Kurukaya, G., Kaya, K., Avanoğlu, O., ve Topal, T. (2017). Özel yetenekli bireylerin eğitimi strateji ve uygulama kılavuzu [Education of specially gifted individuals’ strategy and implementation guide]. MEB Özel Eğitim ve Rehberlik Hizmetleri Genel Müdürlüğü. https://orgm.meb.gov.tr/meb_iys_dosyalar/2013_11/25034903_zelyeteneklibireylerineitimistratejiveuygulamaklavuzu.pdf
7. BİLSEM Online (2023a). Sıkça sorulan sorular: BİSEM sınav soruları yeteneğe göre değişir mi? [Frequently asked questions: Do BİLSEM exam questions vary depending on ability?]. https://www.bilsemonline.com/sss
8. BİLSEM Online (2023b). BNV Zeka Testi Nedir? [What is BNV Intelligence Test?]. https://bilsemonline.com/blog/bnv-zeka-testi-nedir

9. Choi, J., & Zhang, X. (2019). Computerized item modeling practices using computer adaptive formative assessment automatic item generation system: A tutorial. The Quantitative Methods for Psychology, 15(3), 214-225. https://doi.org/10.20982/tqmp.15.3.p214
10. Cohen, R.J., & Swerdlik, M.E. (2015). Psychological testing and assessment. McGraw-Hill Education.
11. Cooper, L.A. (1975). Mental rotation of random two-dimensional shapes. Cognitive Psychology, 7(1), 20-43. https://doi.org/10.1016/0010-0285(75)90003-1
12. DeThorne, L.S. & Schaefer, B.A. (2004). A guide to child nonverbal IQ measures. American Journal of Speech-Language Psychology, 13(4), 275-290. https://doi.org/10.1044/1058-0360(2004/029)
13. Embretson, S.E., & Kingston, N.M. (2018). Automatic item generation: A more efficient process for developing mathematics achievement items? Journal of Educational Measurement, 55(1), 112-131. https://doi.org/10.1111/jedm.12166
14. Embretson, S., & Yang, X. (2007). 23 Automatic item generation and cognitive psychology. Handbook of statistics, 26, 747-768. https://doi.org/10.1016/S0169-7161(06)26023-1
15. Falcão, F., Costa, P., & Pêgo, J.M. (2022). Feasibility assurance: a review of automatic item generation in medical assessment. Advances in Health Sciences Education, 27(2), 405-425. https://doi.org/10.1007/s10459-022-10092-z
16. Gibbons, A., & Warne, R.T. (2019). First publication of subtests in the Stanford-Binet 5, WAIS-IV, WISC V, and WPPSI IV. Intelligence, 75, 9 18. https://doi.org/10.1016/j.intell.2019.02.005
17. Gierl, M.J., & Haladyna, T. (2012). Automatic item generation: Theory and practice. Routledge.
18. Gierl, M.J., & Lai, H. (2012). The role of item models in automatic item generation. International Journal of Testing, 12(3), 273 298. https://doi.org/10.1080/15305058.2011.635830
19. Gierl, M.J., & Lai, H. (2013). Instructional topics in educational measurement (ITEMS) module: Using automated processes to generate test items. Educational Measurement: Issues and Practice, 32(3), 36-50. https://doi.org/10.1111/emip.12018
20. Gierl, M.J. & Lai, H. (2016). Automatic item generation. In S. Lane, M. Raymond, & T. Haladyna (Eds.), Handbook of test development (2nd edition, pp. 410-429). Routledge.
21. Gierl, M.J., Ball, M.M., Vele, V., & Lai, H. (2015). A method for generating nonverbal reasoning items using n-layer modeling. In Computer Assisted Assessment. Research into E-Assessment: 18th International Conference, CAA 2015, Zeist, The Netherlands, June 22–23, 2015. https://doi.org/10.1007/978-3-319-27704-2_2
22. Gierl, M., Lai, H., & Tanygin, V. (2021). Advanced methods in automatic item generation. Routledge.
23. Haladyna, T. M., Downing, S. M. & Rodriguez, M. C. (2002) A review of multiple-choice item-writing guidelines for classroom assessment, Applied Measurement in Education, 15(3), 309-333, https://doi.org/10.1207/S15324818AME1503_5
24. Hausknecht, J.P., Halpert, J.A., Di Paolo, N.T., & Moriarty Gerrard, M.O. (2007). Retesting in selection: A meta-analysis of coaching and practice effects for tests of cognitive ability. Journal of Applied Psychology, 92(2), 373–385. https://doi.org/10.1037/0021-9010.92.2.373
25. Horn, J.L., & Cattell, R.B. (1966). Refinement and test of the theory of fluid and crystallized general intelligences. Journal of Educational Psychology, 57(5), 253 270. https://doi.org/10.1037/h0023816
26. Karasar, N. (2022). Bilimsel araştırma yöntemleri (37. Basım). Nobel Yayıncılık.
27. Kemer, B., & Çakan, M. (2020). Examining the validity of the psychological scales frequently used in guidance and research centers with respect to measurement standards of validity. Journal of Research in Education and Society, 7(1), 323 348. https://dergipark.org.tr/en/pub/etad/issue/55359/731549
28. Kocagül, M., & Çoban, G.Ü. (2022). An evaluation on science teachers' scientific reasoning skills. Cumhuriyet International Journal of Education, 11(2), 361 373. https://doi.org/10.30703/cije.1017938
29. Kosh, A.E., Simpson, M.A., Bickel, L., Kellogg, M., & Sanford‐Moore, E. (2019). A cost–benefit analysis of automatic item generation. Educational Measurement: Issues and Practice, 38(1), 48-53. https://doi.org/10.1111/emip.12237
30. Kurnaz, A., & Ekici, S.G. (2020). BİLSEM tanılama sürecinde kullanılan zeka testlerinin psikolojik danışmanların ve BİLSEM öğretmenlerinin görüşlerine göre değerlendirilmesi [Evaluation of intelligence tests used in BİLSEM diagnostic process according to the opinions of psychological counselors and BİLSEM Teachers]. Çocuk ve Medeniyet, 5(10), 365-399. https://dergipark.org.tr/en/pub/cm/issue/59377/850922
31. Kurtz, K., Gentner, D., & Gunn, V. (1999). Reasoning. In B. M. Bly & D. E. Rumelhart (Eds), Cognitive science, pp. 145-200. California: Academic Press.
32. Lai, H., Gierl, M.J., Byrne, B.E., Spielman, A.I., & Waldschmidt, D.M. (2016). Three modeling applications to promote automatic item generation for examinations in dentistry. Journal of Dental Education, 80(3), 339-347. PMID: 26933110.
33. Lawson, A.E. (2004). The nature and development of scientific reasoning: A synthetic view. International Journal of Science and Mathematics Education, 2(3), 307-338. https://doi.org/10.1007/s10763-004-3224-2
34. Leighton, J.P. (2012). Learning sciences, cognitive models, and automatic item generation. In M.J. Gierl, & T.M. Haladyna (Eds), Automatic item generation: Theory and practice, pp. 121-135. Routledge.
35. Lewis, J.D., DeCamp-Fritson, S.S., Ramage, J.C., McFarland, M.A., & Archwamety, T. (2007). Selecting for ethnically diverse children who may be gifted using Raven’s Standard Progressive Matrices and Naglieri Nonverbal Abilities Test. Multicultural Education, 15(1), 38-42.
36. Lohman, D.F., & Hagen, E. (2003). Interpretive guide for teachers and counselors: cognitive abilities test Form 6-all levels. ITASCA, Illinois: Riverside Publishing.
37. Mercan, Z. (2021). Studies on early childhood reasoning skills in Turkey. Journal of Muallim Rıfat Faculty of Education, 3(2), 104-120.
38. Ministry of National Education (MoNE) (2015). Bilim ve Sanat Merkezleri Yönergesi [Science and Art Centers Directive]. MoNE General Directorate of Special Education and Guidance Services. https://orgm.meb.gov.tr/meb_iys_dosyalar/2015_10/26091626_blsemkilavuz26.10.2015.pdf
39. Ministry of National Education (MoNE) (2021). Bilim ve Sanat Merkezleri Yönergesi [Science and Art Centers Directive]. MoNE General Directorate of Special Education and Guidance Services]. https://orgm.meb.gov.tr/meb_iys_dosyalar/2021_12/30144032_2021 2022_YILI_BILIM_VE_SANAT_MERKEZLERI_OGRENCI_TANILAMA_VE_YERLESTIRME_KILAVUZU.pdf
40. Ministry of National Education (MoNE) (2022a). Bilim ve Sanat Merkezleri Yönergesi [Science and Art Centers Directive]. MoNE General Directorate of Special Education and Guidance Services. https://orgm.meb.gov.tr/meb_iys_dosyalar/2016_10/07031350_bilsem_yonergesi.pdf
41. Ministry of National Education (MoNE) (2022b). Bilim ve Sanat Merkezleri öğrenci tanılama ve yerleştirme kılavuzu [Science and Art Centers student identification and placement guide]. MoNE General Directorate of Special Education and Guidance Services. https://orgm.meb.gov.tr/www/bilsem-ogrenci-tanilama-ve-yerlestirme-kilavuzu-yayimlandi/icerik/2154
42. Mullin, I.V.S., Martin, M.O., & Foy, P. (2005). IEA’s TIMSS 2003 international report on achievement in the mathematics cognitive domains. TIMSS & PIRLS Internatonal study Center. Lynch School of Education, Boston College. https://files.eric.ed.gov/fulltext/ED494652.pdf
43. Naglieri, J.A., & Ford, D.Y. (2005). Increasing minority children’s representation in gifted education: A response to Lohman. Gifted Child Quarterly, 49(1), 29 36. https://doi.org/10.1177/001698620504900104
44. Nolte, N., Schmitz, F., Fleischer, J., Bungart, M., & Leutner, D. (2022). Rotational complexity in mental rotation around cardinal and skewed rotation axes. Intelligence, 91, 101626. https://doi.org/10.1016/j.intell.2022.101626
45. Pylyshyn, Z. W. (1979). The rate of “mental rotation” of images: A test of a holistic analogue hypothesis. Memory & Cognition, 7(1), 19-28. https://doi.org/10.3758/BF03196930
46. Ryoo, J.H., Park, S., Suh, H., Choi, J., & Kwon, J. (2022). Development of a new measure of cognitive ability using automatic item generation and its psychometric properties. SAGE Open, 1-13. https://doi.org/10.1177/21582440221095016
47. Sak, U., Sezerel, B.B., Dulger, E., Sozel, K., & Ayas, M.B. (2019). Validity of the Anadolu-Sak Intelligence Scale in the identification of gifted students. Psychological Test and Assessment Modeling, 61(3), 263-283. https://psycnet.apa.org/record/2020-53108-001
48. Sayın, A., & Gierl, M.J. (2023). Automatic item generation for online measurement and evaluation: Turkish literature items. International Journal of Assessment Tools in Education, 10(2), 218-231. https://doi.org/10.21449/ijate.1249297
49. Shin, E. (2021). Automated item generation by combining the non-template and template-based approaches to generate reading inference test items. [Doctoral dissertation, University of Alberta]. Education and Research Archive. https://doi.org/10.7939/r3-75wr-hc80
50. Sinharay, S., & Johnson, M. (2005). Analysis of data from an admissions test with item models. ETS Research Report Series, 2005(1), 1 32. https://files.eric.ed.gov/fulltext/EJ1111287.pdf
51. Tamul, Ö.F., Sezerel, B.B., Sak, U., & Karabacak, F. (2020). Social validity study of the Anadolu-SAK intelligence scale (ASIS). PAU Journal of Education, 49, 393-412. https://doi.org/10.9779/pauefd.575479
52. Weiss, L.C., Saklofske, D.H., Holdnack, J.A., & Prifitera, A. (2016). WISC-V: Advances in the assessment of intelligence. In L.G. Weiss, D.H. Saklofske, J.A. Holdnack, & A. Prifitera (Eds.), WISC-V assessment and interpretation: Scientist-practitioner perspectives (pp. 3–23). Elsevier Academic Press. https://doi.org/10.1016/B978-0-12-404697-9.00001-7