Use of eye tracking in instrument construction and/or validity studies: Scoping Review

Year 2025, Volume: 16 Issue: 2, 88 - 114, 30.06.2025

Francisco Sousa , André Castelo Branco Soares Francisco Lucas De Lima Fontes Glauber Cavalcante Oliveira Luana Savana Nascimento De Sousa Arruda José Wicto Pereira Borges

https://doi.org/10.21031/epod.1569440

Abstract

Aims to map the evidence on the use of eye tracking in studies on instruments' construction and/or validity. This is a scoping review, based on Joanna Briggs Institute, protocol is registered in the Open Science Framework. Searches were carried out in EMBASE; SCOPUS; Web of Science; MEDLINE/PubMed; CINAHL; LILACS, PsycINFO and gray literature. Eligibility criteria: studies with no time frame or languages, which used eye tracking in the instrument's construction and/or evidence of validity and exclusion: ongoing studies, letters to the editor, abstracts. A total of 26 studies were included in the final sample. Eye tracking was used in a variety of ways to construct and/or validate instruments, with a greater focus on experimental studies, with multiple-choice questionnaires, the item reading process, to check the order and format of questionnaire responses, to evaluate complex questions and improved versions of instruments, response processes, learning styles, asthenopia, cognitive impairment and capacity, discrete choice and instrument reliability, in facial emotion recognition. The main metrics collected were fixation rates and duration, areas of interest, dwell time, GazePlot and Heatmap, saccades and refixation rates; dwell time, GazePlot and Heatmap, saccades and refixation rate. Eye tracking has been used in investigative approaches based on the validity of multiple-choice questionnaires, adapted and applied in exploratory research, the focus of which is to analyze the option order, format, item writing, response style and cognitive evaluations of the participants' response process.

Keywords

Eye-tracking technology, Eye Movements, Validation Study, Surveys and Questionnaires

Ethical Statement

I declare that all ethical guidelines for the author have been followed. This study did not require approval from an ethics committee as it is a scoping review, whit involves the analysis of secondary data already published and does not include the collection of data directly from human subjects.

Project Number

Not applicable

References

• AERA, APA, & NCME. (2014). Standards for Educational and Psychological Testing. Published by the American Educational Research Association. ISBN 978-0-935302-35-6 https://www.aera.net/Publications/Books/Standards-for-Educational-Psychological-Testing-2014-Edition.
• Akgün, H., & Ünaldı, A. (2022). Investigating cognitive processes in different item formats in reading tests through eye-tracking and verbal protocols. Studies in Language Assessment, 98–129. https://doi.org/10.58379/uixk2220
• Araújo, W. C. (2020). Health information retrieval: construction, models and strategies. ConCI: Convergences in Information Science, 3(2), 100–134. https://doi.org/10.33467/conci.v3i2.13447
• Arksey, H., & O'Malley, L. (2005). Scoping studies: towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616
• Balam, G. N., & Osório, A. A. C. (2018). Eye Tracking: Possibilities and Challenges of Using Technology in Infant Samples. Psychology - Theory and Practice, 20(1), 168–188. https://doi.org/10.5935/1980-6906/psicologia.v20n1p179-188
• Baltulionis, S., Turenko, V., Vasiljevas, M., Damaševičius, R., & Sidekerskienė, T. (2019). Validation of VARK questionnaire using gaze tracking data. In Proceedings of the CEUR Workshop: International IVUS 2019 Conference on Information Technology: Proceedings of the International Conference on Information Technology, Kaunas, Lithuania, April 25, 2019 (pp. 28-32). CEUR-WS. https://epubl.ktu.edu/object/elaba:42213647/
• Barreto, A. M. (2012). Eye Tracking as a Research Method Applied to Communication Sciences. Revista Comunicando, 1(1), 168–186. Available at: https://revistas.sopcom.pt/index.php/comunicando/article/view/126 https://doi.org/10.58050/comunicando.v1i1.126
• Batista, L. d. S., Kumada, K. M. O., & Benitez, P. (2023). Eye Tracking and inclusive special education: A systematic review. Olhar de Professor, 26, 1–22. https://doi.org/10.5212/olharprofr.v.26.19672.002
• Bax, D. (2015). Using eye-tracking to research the cognitive processes of multinational readers during an IELTS reading test. IELTS Research Reports Online Series. https://ielts.org/researchers/our-research/research-reports/using-eye-tracking-to-research-the-cognitive-processes-of-multinational-readers-during-an-ielts-reading-test
• Bax, S., & Weir, C. J. (2012). Investigating learners’ cognitive processes during a computer-based CAE Reading test. Cambridge Esol: Research Notes, (47), 3-14. https://www.cambridgeenglish.org/images/22669-rv-research-notes-47.pdf
• Bax, S. (2013). The cognitive processing of candidates during reading tests: Evidence from eye-tracking. Language Testing, 30(4), 441–465. https://doi.org/10.1177/0265532212473244
• Beesley, T., Pearson, D., & Le Pelley, M. (2019). Eye Tracking as a Tool for Examining Cognitive Processes. In Biophysical Measurement in Experimental Social Science Research (pp. 1–30). Elsevier. https://doi.org/10.1016/b978-0-12-813092-6.00002-2
• Brito, M. F. S. F., Borém, L. M. A., Messias, R. B., Silveira, M. F., Souza, A. M. V. d., Leite, M. T. d. S., & Rodrigues Neto, J. F. (2018). Development and validation of an assessment instrument for aspects influencing the request for medical exams. Journal of Collective Health, 26(3), 308–317. https://doi.org/10.1590/1414-462x201800030012
• Büschken, J., Böckenholt, U., Otter, T., & Stengel, D. (2021). Better Information from Survey Data: Filtering Out State Dependence Using Eye-Tracking Data. Psychometrika. https://doi.org/10.1007/s11336-021-09814-w
• Caloca-Amber, S., Mauriz, E., & Vázquez-Casares, A. M. (2024). Exploring eye-tracking data as an indicator of situational awareness in nursing students during a cardiorespiratory arrest simulation. Nurse Education in Practice, 103911. https://doi.org/10.1016/j.nepr.2024.103911
• Camargo, M. V. Z. A. (2022). Analysis of eye movement in associative memory tasks for early diagnosis of Alzheimer's disease (Master's dissertation, University of São Paulo, São Paulo). https://www.teses.usp.br/teses/disponiveis/5/5142/tde-03102022-111832/
• Chauliac, M., Catrysse, L., Gijbels, D., & Donche, V. (2020). It is all in the surv-eye: can Eye Tracking data shed light on the internal consistency in self-report questionnaires on cognitive processing strategies? Frontline Learning Research, 8(3), 26–39. https://doi.org/10.14786/flr.v8i3.489
• Çorbacı, E. C., & Kahraman, N. (2022). Latent Growth Modeling of Item Process Data Derived from Eye-tracking Technology: An Experimental Study Investigating Reading Behavior of Examinees When Answering a Multiple-Choice Test Item. Journal of Measurement and Evaluation in Education and Psychology, 13(3), 194-211. https://doi.org/10.21031/epod.1107597
• Durtette, A., Schmid, F., Barrière, S., Obert, A., Lang, J., Raucher-Chéné, D., Gierski, F., Kaladjian, A., & Henry, A. (2023). Facial emotion recognition processes according to schizotypal personality traits: An eye-tracking study. International Journal of Psychophysiology, 190, 60–68. https://doi.org/10.1016/j.ijpsycho.2023.06.006
• Fang, X., Sun, Y., Zheng, X., Wang, X., Deng, X., & Wang, M. (2021). Assessing Deception in Questionnaire Surveys with Eye-Tracking. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.774961
• Ferretti-Rebustini, R. E. D. L. (2023). Psychometrics: Applications in nursing. Latin American Journal of Nursing, 31. https://doi.org/10.1590/1518-8345.0000.3994
• Galesic, M., Tourangeau, R., Couper, M. P., & Conrad, F. G. (2008). Eye-Tracking Data: New Insights on Response Order Effects and Other Cognitive Shortcuts in Survey Responding. Public Opinion Quarterly, 72(5), 892–913. https://doi.org/10.1093/poq/nfn059
• Gallina, M., Eugênio Spers, E., Florêncio de Almeida, L., Zalla, S., & Meneses, J. V. de. (2022). Visual Attention to Design-Congruent Packaging and Consumer Choice: an experiment using eye-tracking. International Journal of Business and Marketing, 7(2), 86–96. https://doi.org/10.18568/ijbmkt.7.2.194
• Graesser, A. C., Louwerse, M. M., & Daniel, F. (2006). Question Understanding Aid (QUAID): A Web Facility that Tests Question Comprehensibility. Public Opinion Quarterly, 70 (1), 3–22. https://doi.org/10.1093/poq/nfj012
• Gobbi, A. G., Merino, E. A. D., Merino, G. S. A. D., & Gontijo, L. A. (2017). Use of Eye Tracking to Obtain Quantitative Measures in Usability Testing: A Study Focused on Measuring Satisfaction. Human Factors in Design, 6(11), 106–125. https://doi.org/10.5965/2316796306112017106
• Höhne, J. K., & Lenzner, T. (2017). New Insights on the Cognitive Processing of Agree/Disagree and Item-Specific Questions. Journal of Survey Statistics and Methodology, 6(3), 401–417. https://doi.org/10.1093/jssam/smx028
• Hughes, D. J. (2018). Psychometric Validity. In The Wiley Handbook of Psychometric Testing (pp. 751–779). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118489772.ch24
• Kim, Y., Lee, M. J., Choi, M., Cho, E., & Ryu, G. W. (2023). Exploring nurses' multitasking in clinical settings using a multimethod study. Scientific Reports, 13(1). https://doi.org/10.1038/s41598-023-32350-9
• Koutsogiorgi, C. C., & Michaelides, M. P. (2022). Response tendencies due to item wording using eye-tracking methodology accounting for individual differences and item characteristics. Behavior Research Methods. https://doi.org/10.3758/s13428-021-01719-x
• Jesus, G. R. d., Rêgo, R. M. d. L., & Souza, V. V. d. (2022). A questão da validade na avaliação educacional brasileira. Ensaio: Avaliação e Políticas Públicas em Educação, 30(114), 52–72. https://doi.org/10.1590/s0104-40362022003002362
• Junior, S. S., Lukasova, K., Carthery-Goulart, M. T., & Sato, J. R. (2021). Student’s Perspective and Teachers’ Metacognition: Applications of Eye-Tracking in Education and Scientific Research in Schools. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.673615
• Lenzner, T., Kaczmirek, L., & Galesic, M. (2011). Seeing Through the Eyes of the Respondent: An Eye-tracking Study on Survey Question Comprehension. International Journal of Public Opinion Research, 23(3), 361-373. https://doi.org/10.1093/ijpor/edq053
• Liu, Z., Zhang, K., Gao, S., Yang, J., & Qiu, W. (2022). Correlation between Eye Movements and Asthenopia: A Prospective Observational Study. Journal of Clinical Medicine, 11(23), 7043. https://doi.org/10.3390/jcm11237043
• Liu, Y., Zhan, P., Fu, Y., Chen, Q., Man, K., & Luo, Y. (2023). Using a multi-strategy eye-tracking psychometric model to measure intelligence and identify cognitive strategy in Raven's advanced progressive matrices. Intelligence, 100, 101782. https://doi.org/10.1016/j.intell.2023.101782
• Luo, Z., & Wang, Y. (2019). Eye-tracking technology in identifying visualizers and verbalizers: data on eye-movement differences and detection accuracy. Data in Brief, 26, 104447. https://doi.org/10.1016/j.dib.2019.104447
• Obadă, D.-R. (2021). Pretesting flow questionnaire design using eye-tracking: An exploratory study. Argumentum. Journal of the Seminar of Discursive Logic, Argumentation Theory and Rhetoric, 19(1), 97–102. https://philpapers.org/archive/OBAPFQ.pdf
• Maddox, B., Bayliss, A. P., Fleming, P., Engelhardt, P. E., Edwards, S. G., & Borgonovi, F. (2018). Observing response processes with Eye Tracking in international large-scale assessments: evidence from the OECD PIAAC assessment. European Journal of Psychology of Education, 33(3), 543–558. https://doi.org/10.1007/s10212-018-0380-2
• Maia, C. A. A.S, Negreiros, F. D. S., & Moreira, T. M. M. (2023). Evaluative instruments of patients' knowledge about systematic arterial hypertension and diabetes mellitus: a scoping review. Nurse Education in Practice, 103838. https://doi.org/10.1016/j.nepr.2023.103838
• Man, K., Harring, J. R., & Zhan, P. (2022). Bridging Models of Biometric and Psychometric Assessment: A Three-Way Joint Modeling Approach of Item Responses, Response Times, and Gaze Fixation Counts. Applied Psychological Measurement, 46(5), 361-381. https://doi.org/10.1177/01466216221089344
• Milte, R., Crocker, M., Lay, K., Ratcliffe, J., Mulhern, B., Norman, R., Viney, R., & Khadka, J. (2023). Feasibility of self-reported health related quality of life assessment with older people in residential care: insights from the application of Eye Tracking technology. Quality of Life Research. https://doi.org/10.1007/s11136-023-03488-w
• Monteiro, S. D., Rodas, C. M., & Vidotti, S. A. B. G. (2020). A busca e o Eye Tracking: um olhar semiótico sobre o knowledge graph. Em Questão, 26(1), 304–326. https://doi.org/10.19132/1808-5245261.304-326
• Neuert, C. E. (2019). How Effective Are Eye-Tracking Data in Identifying Problematic Questions? Social Science Computer Review, 38(6), 793–802. https://doi.org/10.1177/0894439319834289
• Padilla, J. L., & Benítez, I. (2014). Validity evidence based on response processes. Psicothema, 26(1), 136–144. https://doi.org/10.7334/psicothema2013.259
• Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., ... Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, n71. https://doi.org/10.1136/bmj.n71
• Pauszek, J. R. (2022). An Introduction to Eye Tracking in Human Factors Healthcare Research and Medical Device Testing. Human Factors in Healthcare, 100031. https://doi.org/10.1016/j.hfh.2022.100031
• Peters, M., Godfrey, C., McInerney, P., Munn, Z., Trico, A., & Khalil, H. (2020). Chapter 11: Scoping Reviews. In JBI Manual for Evidence Synthesis. JBI. https://doi.org/10.46658/jbimes-20-12
• Ratcliffe, J., Lay, K., Crocker, M., Engel, L., Milte, R., Hutchinson, C., Khadka, J., Whitehurst, D. G. T., Mulhern, B., Viney, R., & Norman, R. (2023). Unravelling the Self-Report Versus Proxy-Report Conundrum for Older Aged Care Residents: Findings from a Mixed-Methods Study. The Patient - Patient-Centered Outcomes Research. https://doi.org/10.1007/s40271-023-00655-6
• Reppold, C.T., Zanini, DS., & Noronha, A. P. (2019). What is psychological assessment? In M.N. Baptista et al. (Eds), Compendium of psychological Assesmment. Petropolis, RJ: Vozes. ISBN 978-85-326-6360-3. https://abrir.link/Y5t4J
• Requena, F. C. G. (2022). Visual analysis of reading patterns in Portuguese texts through Eye Tracking (Master's dissertation, FEI University Center, São Bernardo do Campo). Available at: https://repositorio.fei.edu.br/items/61e221d4-a51f-4747-8281-b0f4de3b8a04
• Rivu, R., Abdrabou, Y., Abdelrahman, Y., Pfeuffer, K., Kern, D., Neuert, C., Buschek, D., & Alt, F. (2021). Did you Understand this? In ETRA '21: 2021 Symposium on Eye Tracking Research and Applications. ACM. https://doi.org/10.1145/3448018.3458018
• Rodrigues, E. S. (2021). Analysis of eye-tracking data in game evaluation: A case study (Master's dissertation). Federal University of Campina Grande. Available from http://dspace.sti.ufcg.edu.br:8080/jspui/handle/riufcg/21364
• Tontini, J. (2021). Information processing and Eye Tracking in pre-test research (Master's dissertation, Postgraduate Program in Administration, Federal University of Santa Maria). Santa Maria, RS: Universidade Federal de Santa Maria. Retrieved from https://repositorio.ufsm.br/handle/1/25677
• Wang, K., Barr, C., Norman, R., George, S., Whitehead, C., & Ratcliffe, J. (2020). Using Eye-Tracking Technology with Older People in Memory Clinics to Investigate the Impact of Mild Cognitive Impairment on Choices for EQ-5D-5L Health States Preferences. Applied Health Economics and Health Policy, 19(1), 111–121. https://doi.org/10.1007/s40258-020-00588-3
• Yaneva, V., Clauser, B. E., Morales, A., & Paniagua, M. (2021). Using Eye‐Tracking Data as Part of the Validity Argument for Multiple‐Choice Questions: A Demonstration. Journal of Educational Measurement, 58(4), 515–537. https://doi.org/10.1111/jedm.12304
• Yaneva, V., Clauser, B. E., Morales, A., & Paniagua, M. (2022). Assessing the validity of test scores using response process data from an eye-tracking study: a new approach. Advances in Health Sciences Education. https://doi.org/10.1007/s10459-022-10107-9