Evaluating the Psychometric Characteristics of Generated Visual Reading Comprehension Items

Abstract

Reading comprehension, a crucial skill in today's information-rich environment, extends beyond text to include visual elements. Manual creation of visual reading comprehension items poses challenges, necessitating an innovative approach. This situation has led to the exploration of Automatic Item Generation (AIG) as a solution. This study aims to demonstrate the use of AIG for the creation of visual reading comprehension items. By developing cognitive and item models through expert input and utilizing computer algorithms for item generation, the study seeks to provide a time-efficient and reliable alternative for item writers. The field test involved 1,380 8th-grade students to evaluate the psychometric properties of the generated visual reading comprehension items. The AIG process starts with expert insights to develop cognitive and item models. Computer algorithms are then employed for AIG. The study utilizes a diverse sample of 8th-grade students for field testing, assessing the psychometric properties of the generated items. Field test results indicate the potential of AIG in efficiently producing a substantial item pool for visual reading comprehension. The generated items exhibit consistent difficulty levels (0.58 to 0.66), ensuring an appropriate challenge for students. High item discrimination (0.48 to 0.69) effectively distinguishes between students with varying visual reading comprehension skills. Item-total correlations (0.40 to 0.57) further validate the quality and validity of the generated items. The automated process yields efficient results in terms of item difficulty and discrimination, emphasizing the potential of AIG for high-quality assessment of visual reading comprehension items.

Keywords

• visual reading comprehension
• automatic item generation
• high-stake tests
• parallel test

References

1. Adedoyin, O. O., Nenty, H., & Chilisa, B. (2008). Investigating the invariance of item difficulty parameter estimates based on CTT and IRT. Educational Research and Reviews, 3(3), 83.
2. Adji, T. B., Pribadi, F. S., Prabowo, H. E., Rosnawati, R., & Wijaya, A. (2018). Generating Parallel Mathematic Items Using Automatic Item Generation. ICEAP 2019, 1(1), 89-93.
3. Aiken, L. R. (1979). Relationships between the item difficulty and discrimination indexes. Educational and Psychological Measurement, 39(4), 821-824.
4. Al-Hameed, F. and Al-Shuair, M. (2019). The effectiveness of using digital stories (on internet) to improve the literal, organizational and inferential reading comprehension skills of English as a second language. Journal of Research in Curriculum Instruction and Educational Technology, 5(3), 45-81.
5. Aloqaili, A. S. (2012). The relationship between reading comprehension and critical thinking: A theoretical study. Journal of King Saud University-Languages and Translation, 24(1), 35-41.
6. Amin, M. (2019). Developing reading skills through effective reading approaches. International Journal of Social Science and Humanities, 4(1), 35-40.
7. Arendasy, M. E., & Sommer, M. (2012). Using automatic item generation to meet the increasing item demands of high-stakes educational and occupational assessment. Learning and individual differences, 22(1), 112-117.
8. Barnes, M. A. (2015). What do models of Reading comprehension and its development have to contribute to a science of comprehension instruction and assessment for adolescents? Improving reading comprehension of middle and high school students, 1-18.

9. Boonen, A. J., de Koning, B. B., Jolles, J., & Van der Schoot, M. (2016). Word problem solving in contemporary math education: A plea for reading comprehension skills training. Frontiers in psychology, 7, 191.
10. Brennan, R. L. (1972). A generalized upper-lower item discrimination index. Educational and Psychological Measurement, 32(2), 289-303.
11. Brenner, M. H. (1964). Test difficulty, reliability, and discrimination as functions of item difficulty order. Journal of Applied Psychology, 48(2), 98.
12. Brevik, L. M. (2019). Explicit reading strategy instruction or daily use of strategies? Studying the teaching of reading comprehension through naturalistic classroom observation in English L2. Reading and writing, 32(9), 2281-2310.
13. Cahalan, C., Mandinach, E. B., & Camara, W. J. (2002). Predictive Validity of SAT® I: Reasoning Test for Test-Takers with Learning Disabilities and Extended Time Accommodations. Research Report No. 2002-5. ETS RR-02-11. College Entrance Examination Board.
14. Caldwell, D. J., & Pate, A. N. (2013). Effects of question formats on student and item performance. American journal of pharmaceutical education, 77(4).
15. Chandran, Y., & Shah, P. M. (2019). Identifying learners’ difficulties in ESL reading comprehension. Creative Education, 10(13), 3372-3384.
16. Cohen, A. D., & Upton, T. A. (2006). Strategies in responding to the new TOEFL reading tasks. ETS Research Report Series, 2006(1), i-162.
17. Cornoldi, C., & Oakhill, J. V. (2013). Reading comprehension difficulties: Processes and intervention. Routledge.
18. Daly, A., & Unsworth, L. (2011). Analysis and comprehension of multimodal texts. Australian Journal of Language and Literacy, The, 34(1), 61-80.
19. Duncan, L., McGeown, S., Griffiths, Y., Stothard, S., & Dobai, A. (2015). Adolescent reading skill and engagement with digital and traditional literacies as predictors of reading comprehension. British Journal of Psychology, 107(2), 209-238.
20. Eason, S., Goldberg, L., Young, K., Geist, M., & Cutting, L. (2012). Reader–text interactions: how differential text and question types influence cognitive skills needed for reading comprehension. Journal of Educational Psychology, 104(3), 515-528.
21. Embretson, S., & Yang, X. (2006). 23 Automatic item generation and cognitive psychology. Handbook of statistics, 26, 747-768.
22. Embretson, S. E., & Kingston, N. M. (2018). Automatic item generation: A more efficient process for developing mathematics achievement items? Journal of Educational Measurement, 55(1), 112-131.
23. Falcão, F., Costa, P., & Pêgo, J. M. (2022). Feasibility assurance: a review of automatic item generation in medical assessment. Advances in Health Sciences Education, 27(2), 405-425.
24. Falcão, F., Pereira, D. M., Gonçalves, N., De Champlain, A., Costa, P., & Pêgo, J. M. (2023). A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation. Advances in Health Sciences Education, 1-25.
25. Fielding, L. G., & Pearson, P. D. (1994). Reading Comprehension: What Works. Educational leadership, 51(5), 62-68.
26. Freedle, R., & Kostin, I. (1991). The prediction of SAT reading comprehension item difficulty for expository prose passages. ETS Research Report Series, 1991(1), i-52.
27. Fu, Y., Choe, E. M., Lim, H., & Choi, J. (2022). An Evaluation of Automatic Item Generation: A Case Study of Weak Theory Approach. Educational Measurement: Issues and Practice.
28. Gierl, M., Lai, H., & Zhang, X. (2019). Automatic item generation. In Advanced Methodologies and Technologies in Modern Education Delivery (pp. 192-203). IGI Global.
29. Gierl, M. J., & Haladyna, T. M. (2012a). Automatic item generation: An introduction. In Automatic item generation (pp. 13-22). Routledge.
30. Gierl, M. J., & Haladyna, T. M. (2012b). Automatic item generation: Theory and practice. Routledge.
31. Gierl, M. J., & Lai, H. (2012). The role of item models in automatic item generation. International journal of testing, 12(3), 273-298.
32. Gierl, M. J., & Lai, H. (2018). Using Automatic Item Generation to Create Solutions and Rationales for Computerized Formative Testing. Applied Psychological Measurement, 42(1), 42-57. https://doi.org/10.1177/0146621617726788
33. Gierl, M. J., Lai, H., Pugh, D., Touchie, C., Boulais, A.-P., & De Champlain, A. (2016). Evaluating the psychometric characteristics of generated multiple-choice test items. Applied Measurement in Education, 29(3), 196-210.
34. Gierl, M. J., Lai, H., & Turner, S. R. (2012). Using automatic item generation to create multiple-choice test items. Medical Education, 46(8), 757-765. https://doi.org/10.1111/j.1365-2923.2012.04289.x
35. Glenberg, A. M., & Langston, W. E. (1992). Comprehension of illustrated text: Pictures help to build mental models. Journal of memory and language, 31(2), 129-151.
36. Gorin, J. S. (2005). Manipulating processing difficulty of reading comprehension questions: The feasibility of verbal item generation. Journal of Educational Measurement, 42(4), 351-373.
37. Gorin, J. S., & Embretson, S. E. (2012). Using cognitive psychology to generate items and predict item characteristics. In Automatic item generation (pp. 146-166). Routledge.
38. Harrison, P., Collins, T., & Müllensiefen, D. (2017). Applying modern psychometric techniques to melodic discrimination testing: Item response theory, computerised adaptive testing, and automatic item generation. Scientific Reports, 7(1), 1-18.
39. Hill, C. A. (2003). Reading the visual in college writing classes. In Intertexts (pp. 134-159). Routledge.
40. Holling, H., Bertling, J. P., & Zeuch, N. (2009). Automatic item generation of probability word problems. Studies in Educational Evaluation, 35(2-3), 71-76.
41. Hosseini, E., Khodaei, F. B., Sarfallah, S., & Dolatabadi, H. R. (2012). Exploring the relationship between critical thinking, reading comprehension and reading strategies of English university students. World Applied Sciences Journal, 17(10), 1356-1364.
42. Hoyt, L. (1992). Many ways of knowing: Using drama, oral interactions, and the visual arts to enhance reading comprehension. The Reading Teacher, 45(8), 580-584.
43. Irvine, S. H., & Kyllonen, P. C. (2013). Item generation for test development. Routledge.
44. Khasawneh, M. A. S., & Al-Rub, M. O. A. (2020). Development of reading comprehension skills among the students of learning disabilities. Universal Journal of Educational Research, 8(11), 5335-5341.
45. Kinniburgh, L. H., & Shaw, E. L. (2009). Using question-answer relationships to build: Reading comprehension in science. Science Activities, 45(4), 19-28.
46. Klingner, J. K., Vaughn, S., & Boardman, A. (2015). Teaching reading comprehension to students with learning difficulties. Guilford Publications.
47. Kosh, A. E., Simpson, M. A., Bickel, L., Kellogg, M., & Sanford‐Moore, E. (2019). A cost–benefit analysis of automatic item generation. Educational Measurement: Issues and Practice, 38(1), 48-53.
48. Lai, H., Gierl, M. J., Byrne, B. E., Spielman, A. I., & Waldschmidt, D. M. (2016). Three modeling applications to promote automatic item generation for examinations in dentistry. Journal of Dental Education, 80(3), 339-347.
49. Lai, H., Gierl, M. J., Touchie, C., Pugh, D., Boulais, A.-P., & De Champlain, A. (2016). Using automatic item generation to improve the quality of MCQ distractors. Teaching and learning in medicine, 28(2), 166-173.
50. Lervåg, A., Hulme, C., & Melby‐Lervåg, M. (2017). Unpicking the developmental relationship between oral language skills and reading comprehension: it's simple, but complex. Child Development, 89(5), 1821-1838.
51. Li, H., Wang, P., Shen, C., & Hengel, A. v. d. (2019). Visual question answering as reading comprehension. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
52. MEB, (2018). Milli Eğitim Bakanlığı Ortaöğretime Geçiş Yönergesi. https://www.meb.gov.tr/meb_iys_dosyalar/2018_03/26191912_yonerge.pdf
53. Mullis, I., Martin, M., Foy, P., & Hooper, M. (2017). PIRLS 2016 International Results in Reading. Boston College, TIMSS & PIRLS International Study Center. In.
54. Mullis, I. V. S., & Martin, M. O. (Eds.). (2019). PIRLS 2021 Assessment Frameworks. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: https://timssandpirls.bc.edu/pirls2021/frameworks/
55. Nummenmaa, L., Glerean, E., Hari, R., & Hietanen, J. K. (2014). Bodily maps of emotions. Proceedings of the National Academy of Sciences, 111(2), 646-651.
56. O'Neil, K. E. (2011). Reading pictures: Developing visual literacy for greater comprehension. The Reading Teacher, 65(3), 214-223.
57. OECD (2019), "PISA 2018 Reading Framework", in PISA 2018 Assessment and Analytical Framework, OECD Publishing, Paris, https://doi.org/10.1787/5c07e4f1-en.
58. OECD. (2021). 21st-Century Readers. https://doi.org/doi:https://doi.org/10.1787/a83d84cb-en
59. Oliver, K. (2009). An investigation of concept mapping to improve the reading comprehension of science texts. Journal of Science Education and Technology, 18, 402-414.
60. Österholm, M. (2006). Characterizing reading comprehension of mathematical texts. Educational studies in mathematics, 63, 325-346.
61. Sabatini, J. P., O’reilly, T., Halderman, L., & Bruce, K. (2014). Broadening the scope of reading comprehension using scenario-based assessments: Preliminary findings and challenges. LAnnee psychologique, 114(4), 693-723.
62. Santi, K. L., Kulesz, P. A., Khalaf, S., & Francis, D. J. (2015). Developmental changes in reading do not alter the development of visual processing skills: an application of explanatory item response models in grades K-2. Frontiers in psychology, 6, 116.
63. Setiawan, H., Hidayah, I., & Kusumawardani, S. S. (2022). Automatic Item Generation with Reading Passages: A Systematic Literature Review. 2022 8th International Conference on Education and Technology (ICET),
64. Shin, E. (2021). Automated Item Generation by Combining the Non-template and Template-based Approaches to Generate Reading Inference Test Items University of Alberta]. Canada.
65. Shin, J., & Gierl, M. J. (2022). Generating reading comprehension items using automated processes. International journal of testing, 22(3-4), 289-311.
66. Silva, M. and Cain, K. (2015). The relations between lower and higher-level comprehension skills and their role in prediction of early reading comprehension. Journal of Educational Psychology, 107(2), 321-331.
67. Unsworth, L. (2014). Multimodal reading comprehension: Curriculum expectations and large-scale literacy testing practices. Pedagogies: An international journal, 9(1), 26-44.
68. Westwood, P. (2016). Teaching and Learning Difficulties 2nd ed (Vol. 2). Acer Press.
69. Woolley, G., & Woolley, G. (2011). Reading comprehension. Springer.
70. Wyer Jr, R. S., Hung, I. W., & Jiang, Y. (2008). Visual and verbal processing strategies in comprehension and judgment. Journal of Consumer Psychology, 18(4), 244-257.