Latent Growth Modeling of Item Process Data Derived From Eye-tracking Technology: An Experimental Study Investigating Reading Behavior of Examinees When Answering A Multiple-Choice Test Item

Abstract

This study illustrates how eye-tracking data can be translated to “item process data” for multiple-choice test items to study the relationship between subjects’ item responses and choice reading behavior. Several modes of analysis were used to test the hypothesized added value of using process data to identify choice reading patterns of subjects. In addition to the cross-sectional analises of agggregate measurements derived from the time series eye tracking data, Latent Growth Curve Model Analises were undertaken to test if the the shape of change observed in the sequential choice reading patterns differed for subjects depending on their responses to the item being correct or incorrect. Application data were from an experimental study and included seventy-one subjects’ responses to two multiple-choice test items measuring reading comprehension ability in English as a second language. Analyses were carried out for one item at a time. For each item, first, each subject’s recorded eye movements were coded into a set of Area of Interests (AOIs), segmenting the lines in the stem and the individual choices. Next, each subject’s fixation times on the AOIs were time stamped into seconds, indicating when and in what order each subject’s gaze had fixated on each AOI until a choice was marked as the correct answer, which ended the item encounter. A set of nested Latent Growth Curve models were considered for the choice-related AOIs to deliniate if distinct choice-process sequences were evident for correct and incorrect respoders. Model fit indices, random intercepts, slopes, and residuals were computed using the mean log fixation times over item encounter time. The results show that the LGM with the best model fit indicies, for both items, was the quadratic model using response variable as a covariate. Albeit limited due to the two-item – seventy-one subjects experimental setting of the study, the findings are promising and show that utilizing item-level process data can be very useful for defining distinct choice processing (task-oriented reading) patterns of examinees. Over all, the results warrant further study of choice derived AOIs using longitudinal statistical models. It is argued that, the screening methodology desribed in this study can be a useful tool to investigate speededness, distractor functioning, or even to flag subjects with irregular choice processing behavior, such as providing a direct mark on a choice, without any significant reading activity on any of the choices presented (i.e., whether cheating might have occurred.)

Keywords

• Latent Growth Curve Modeling
• eye-tracking
• reading ability in English
• multiple-choice items

References

1. Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716-723. https://doi.org/10.1109/TAC.1974.1100705
2. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for educational and psychological testing. American Educational Research Association.
3. Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238-246. https://doi.org/10.1037/0033-2909.107.2.238
4. Bentler, P. M. (1995). EQS structural equations program manual (vol. 6). Multivariate software. https://doi.org/10.4236/am.2014.510132
5. Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological bulletin, 88(3), 588-606. https://doi.org/10.1037/0033-2909.88.3.588
6. Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136−162). Sage.
7. Duncan, T. E., Duncan, S. C., & Strycker, L. A. (2013). An introduction to latent variable growth curve modeling: Concepts, issues, and applications. Routledge.
8. Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & Van de Weijer, J. (2011). Eye tracking: A comprehensive guide to methods and measures. OUP Oxford.

9. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118
10. Kane, M., & Mislevy, R. (2017). Validating score interpretations based on response processes. In Validation of score meaning for the next generation of assessments (pp. 11-24). Routledge.
11. Mézière, D. C., Yu, L., Reichle, E., von der Malsburg, T., & McArthur, G. (2021). Using eye-tracking measures to predict reading comprehension. https://doi.org/10.31234/osf.io/v2rdp
12. Muthén, B. (2001). Second-generation structural equation modeling with a combination of categorical and continuous latent variables: New opportunities for latent class–latent growth modeling. In L. M. Collins & A. G. Sayer (Eds.), New methods for the analysis of change (pp. 291–322). American Psychological Association. https://doi.org/10.1037/10409-010
13. Öğrenci Seçme ve Yerleştirme Merkezi (2018). Retrieved on February 25, from https://www.osym.gov.tr/TR,15313/2018-yds-sonbahar-donemi-temel-soru-kitapciklarinin-yayimlanmasi--10.html
14. Paulson, E. J., & Henry, J. (2002). Does the Degrees of Reading Power assessment reflect the reading process? An eye-movement examination. Journal of Adolescent & Adult Literacy, 46(3), 234-244. https://link.gale.com/apps/doc/A94123361/LitRC?u=anon~924799ab&sid=bookmark- LitRC&xid=2cf242f9
15. Preacher, K. J., Wichman, A. L., MacCallum, R. C., & Briggs, N. E. (2008). Latent growth curve modeling (No. 157). Sage.
16. Raney, G. E., Campbell, S. J., & Bovee, J. C. (2014). Using eye movements to evaluate the cognitive processes involved in text comprehension. JoVE (Journal of Visualized Experiments), 83, e50780. https://doi.org/10.3791/50780
17. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3), 372-422. https://doi.org/10.1037/0033-2909.124.3.372
18. Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461-464. https://doi.org/10.1214/aos/1176344136
19. Solheim, O. J., & Uppstad, P. H. (2011). Eye-tracking as a tool in process-oriented reading test validation. International Electronic Journal of Elemantary Education, 4(1), 153-168. https://www.iejee.com/index.php/IEJEE/article/view/218
20. Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25(2), 173-180. https://doi.org/10.1207/s15327906mbr2502_4
21. Tai, R. H., Loehr, J. F., & Brigham, F. J. (2006). An exploration of the use of eye‐gaze tracking to study problem‐ solving on standardized science assessments. International Journal of Research & Method in Education, 29(2), 185-208. https://doi.org/10.1080/17437270600891614
22. Tsai, M. J., Hou, H. T., Lai, M. L., Liu, W. Y., & Yang, F. Y. (2012). Visual attention for solving multiple-choice science problem: An eye-tracking analysis. Computers & Education, 58(1), 375-385. https://doi.org/10.1016/j.compedu.2011.07.012
23. Willett, J. B., & Sayer, A. G. (1994). Using covariance structure analysis to detect correlates and predictors of individual change. Psychological Bulletin, 110, 268-290. https://doi.org/10.1037/0033-2909.116.2.363
24. Yaneva, V., Clauser, B. E., Morales, A., & Paniagua, M. (2022). Assessing the validity of test scores using response process data from an eye-tracking study: A new approach. Advances in Health Sciences Education, Online First. https://doi.org/10.1007/s10459-022-10107-9