Use of eye tracking in instrument construction and/or validity studies: Scoping Review

Abstract

Aims to map the evidence on the use of eye tracking in studies on instruments' construction and/or validity. This is a scoping review, based on Joanna Briggs Institute, protocol is registered in the Open Science Framework. Searches were carried out in EMBASE; SCOPUS; Web of Science; MEDLINE/PubMed; CINAHL; LILACS, PsycINFO and gray literature. Eligibility criteria: studies with no time frame or languages, which used eye tracking in the instrument's construction and/or evidence of validity and exclusion: ongoing studies, letters to the editor, abstracts. A total of 26 studies were included in the final sample. Eye tracking was used in a variety of ways to construct and/or validate instruments, with a greater focus on experimental studies, with multiple-choice questionnaires, the item reading process, to check the order and format of questionnaire responses, to evaluate complex questions and improved versions of instruments, response processes, learning styles, asthenopia, cognitive impairment and capacity, discrete choice and instrument reliability, in facial emotion recognition. The main metrics collected were fixation rates and duration, areas of interest, dwell time, GazePlot and Heatmap, saccades and refixation rates; dwell time, GazePlot and Heatmap, saccades and refixation rate. Eye tracking has been used in investigative approaches based on the validity of multiple-choice questionnaires, adapted and applied in exploratory research, the focus of which is to analyze the option order, format, item writing, response style and cognitive evaluations of the participants' response process.

Keywords

• Eye-tracking technology
• Eye Movements
• Validation Study
• Surveys and Questionnaires

Project Number

Not applicable

Ethical Statement

I declare that all ethical guidelines for the author have been followed. This study did not require approval from an ethics committee as it is a scoping review, whit involves the analysis of secondary data already published and does not include the collection of data directly from human subjects.

References

1. AERA, APA, & NCME. (2014). Standards for Educational and Psychological Testing. Published by the American Educational Research Association. ISBN 978-0-935302-35-6 https://www.aera.net/Publications/Books/Standards-for-Educational-Psychological-Testing-2014-Edition.
2. Akgün, H., & Ünaldı, A. (2022). Investigating cognitive processes in different item formats in reading tests through eye-tracking and verbal protocols. Studies in Language Assessment, 98–129. https://doi.org/10.58379/uixk2220
3. Araújo, W. C. (2020). Health information retrieval: construction, models and strategies. ConCI: Convergences in Information Science, 3(2), 100–134. https://doi.org/10.33467/conci.v3i2.13447
4. Arksey, H., & O'Malley, L. (2005). Scoping studies: towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616
5. Balam, G. N., & Osório, A. A. C. (2018). Eye Tracking: Possibilities and Challenges of Using Technology in Infant Samples. Psychology - Theory and Practice, 20(1), 168–188. https://doi.org/10.5935/1980-6906/psicologia.v20n1p179-188
6. Baltulionis, S., Turenko, V., Vasiljevas, M., Damaševičius, R., & Sidekerskienė, T. (2019). Validation of VARK questionnaire using gaze tracking data. In Proceedings of the CEUR Workshop: International IVUS 2019 Conference on Information Technology: Proceedings of the International Conference on Information Technology, Kaunas, Lithuania, April 25, 2019 (pp. 28-32). CEUR-WS. https://epubl.ktu.edu/object/elaba:42213647/
7. Barreto, A. M. (2012). Eye Tracking as a Research Method Applied to Communication Sciences. Revista Comunicando, 1(1), 168–186. Available at: https://revistas.sopcom.pt/index.php/comunicando/article/view/126 https://doi.org/10.58050/comunicando.v1i1.126
8. Batista, L. d. S., Kumada, K. M. O., & Benitez, P. (2023). Eye Tracking and inclusive special education: A systematic review. Olhar de Professor, 26, 1–22. https://doi.org/10.5212/olharprofr.v.26.19672.002

9. Bax, D. (2015). Using eye-tracking to research the cognitive processes of multinational readers during an IELTS reading test. IELTS Research Reports Online Series. https://ielts.org/researchers/our-research/research-reports/using-eye-tracking-to-research-the-cognitive-processes-of-multinational-readers-during-an-ielts-reading-test
10. Bax, S., & Weir, C. J. (2012). Investigating learners’ cognitive processes during a computer-based CAE Reading test. Cambridge Esol: Research Notes, (47), 3-14. https://www.cambridgeenglish.org/images/22669-rv-research-notes-47.pdf
11. Bax, S. (2013). The cognitive processing of candidates during reading tests: Evidence from eye-tracking. Language Testing, 30(4), 441–465. https://doi.org/10.1177/0265532212473244
12. Beesley, T., Pearson, D., & Le Pelley, M. (2019). Eye Tracking as a Tool for Examining Cognitive Processes. In Biophysical Measurement in Experimental Social Science Research (pp. 1–30). Elsevier. https://doi.org/10.1016/b978-0-12-813092-6.00002-2
13. Brito, M. F. S. F., Borém, L. M. A., Messias, R. B., Silveira, M. F., Souza, A. M. V. d., Leite, M. T. d. S., & Rodrigues Neto, J. F. (2018). Development and validation of an assessment instrument for aspects influencing the request for medical exams. Journal of Collective Health, 26(3), 308–317. https://doi.org/10.1590/1414-462x201800030012
14. Büschken, J., Böckenholt, U., Otter, T., & Stengel, D. (2021). Better Information from Survey Data: Filtering Out State Dependence Using Eye-Tracking Data. Psychometrika. https://doi.org/10.1007/s11336-021-09814-w
15. Caloca-Amber, S., Mauriz, E., & Vázquez-Casares, A. M. (2024). Exploring eye-tracking data as an indicator of situational awareness in nursing students during a cardiorespiratory arrest simulation. Nurse Education in Practice, 103911. https://doi.org/10.1016/j.nepr.2024.103911
16. Camargo, M. V. Z. A. (2022). Analysis of eye movement in associative memory tasks for early diagnosis of Alzheimer's disease (Master's dissertation, University of São Paulo, São Paulo). https://www.teses.usp.br/teses/disponiveis/5/5142/tde-03102022-111832/
17. Chauliac, M., Catrysse, L., Gijbels, D., & Donche, V. (2020). It is all in the surv-eye: can Eye Tracking data shed light on the internal consistency in self-report questionnaires on cognitive processing strategies? Frontline Learning Research, 8(3), 26–39. https://doi.org/10.14786/flr.v8i3.489
18. Çorbacı, E. C., & Kahraman, N. (2022). Latent Growth Modeling of Item Process Data Derived from Eye-tracking Technology: An Experimental Study Investigating Reading Behavior of Examinees When Answering a Multiple-Choice Test Item. Journal of Measurement and Evaluation in Education and Psychology, 13(3), 194-211. https://doi.org/10.21031/epod.1107597
19. Durtette, A., Schmid, F., Barrière, S., Obert, A., Lang, J., Raucher-Chéné, D., Gierski, F., Kaladjian, A., & Henry, A. (2023). Facial emotion recognition processes according to schizotypal personality traits: An eye-tracking study. International Journal of Psychophysiology, 190, 60–68. https://doi.org/10.1016/j.ijpsycho.2023.06.006
20. Fang, X., Sun, Y., Zheng, X., Wang, X., Deng, X., & Wang, M. (2021). Assessing Deception in Questionnaire Surveys with Eye-Tracking. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.774961
21. Ferretti-Rebustini, R. E. D. L. (2023). Psychometrics: Applications in nursing. Latin American Journal of Nursing, 31. https://doi.org/10.1590/1518-8345.0000.3994
22. Galesic, M., Tourangeau, R., Couper, M. P., & Conrad, F. G. (2008). Eye-Tracking Data: New Insights on Response Order Effects and Other Cognitive Shortcuts in Survey Responding. Public Opinion Quarterly, 72(5), 892–913. https://doi.org/10.1093/poq/nfn059
23. Gallina, M., Eugênio Spers, E., Florêncio de Almeida, L., Zalla, S., & Meneses, J. V. de. (2022). Visual Attention to Design-Congruent Packaging and Consumer Choice: an experiment using eye-tracking. International Journal of Business and Marketing, 7(2), 86–96. https://doi.org/10.18568/ijbmkt.7.2.194
24. Graesser, A. C., Louwerse, M. M., & Daniel, F. (2006). Question Understanding Aid (QUAID): A Web Facility that Tests Question Comprehensibility. Public Opinion Quarterly, 70 (1), 3–22. https://doi.org/10.1093/poq/nfj012
25. Gobbi, A. G., Merino, E. A. D., Merino, G. S. A. D., & Gontijo, L. A. (2017). Use of Eye Tracking to Obtain Quantitative Measures in Usability Testing: A Study Focused on Measuring Satisfaction. Human Factors in Design, 6(11), 106–125. https://doi.org/10.5965/2316796306112017106
26. Höhne, J. K., & Lenzner, T. (2017). New Insights on the Cognitive Processing of Agree/Disagree and Item-Specific Questions. Journal of Survey Statistics and Methodology, 6(3), 401–417. https://doi.org/10.1093/jssam/smx028
27. Hughes, D. J. (2018). Psychometric Validity. In The Wiley Handbook of Psychometric Testing (pp. 751–779). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118489772.ch24
28. Kim, Y., Lee, M. J., Choi, M., Cho, E., & Ryu, G. W. (2023). Exploring nurses' multitasking in clinical settings using a multimethod study. Scientific Reports, 13(1). https://doi.org/10.1038/s41598-023-32350-9
29. Koutsogiorgi, C. C., & Michaelides, M. P. (2022). Response tendencies due to item wording using eye-tracking methodology accounting for individual differences and item characteristics. Behavior Research Methods. https://doi.org/10.3758/s13428-021-01719-x
30. Jesus, G. R. d., Rêgo, R. M. d. L., & Souza, V. V. d. (2022). A questão da validade na avaliação educacional brasileira. Ensaio: Avaliação e Políticas Públicas em Educação, 30(114), 52–72. https://doi.org/10.1590/s0104-40362022003002362
31. Junior, S. S., Lukasova, K., Carthery-Goulart, M. T., & Sato, J. R. (2021). Student’s Perspective and Teachers’ Metacognition: Applications of Eye-Tracking in Education and Scientific Research in Schools. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.673615
32. Lenzner, T., Kaczmirek, L., & Galesic, M. (2011). Seeing Through the Eyes of the Respondent: An Eye-tracking Study on Survey Question Comprehension. International Journal of Public Opinion Research, 23(3), 361-373. https://doi.org/10.1093/ijpor/edq053
33. Liu, Z., Zhang, K., Gao, S., Yang, J., & Qiu, W. (2022). Correlation between Eye Movements and Asthenopia: A Prospective Observational Study. Journal of Clinical Medicine, 11(23), 7043. https://doi.org/10.3390/jcm11237043
34. Liu, Y., Zhan, P., Fu, Y., Chen, Q., Man, K., & Luo, Y. (2023). Using a multi-strategy eye-tracking psychometric model to measure intelligence and identify cognitive strategy in Raven's advanced progressive matrices. Intelligence, 100, 101782. https://doi.org/10.1016/j.intell.2023.101782
35. Luo, Z., & Wang, Y. (2019). Eye-tracking technology in identifying visualizers and verbalizers: data on eye-movement differences and detection accuracy. Data in Brief, 26, 104447. https://doi.org/10.1016/j.dib.2019.104447
36. Obadă, D.-R. (2021). Pretesting flow questionnaire design using eye-tracking: An exploratory study. Argumentum. Journal of the Seminar of Discursive Logic, Argumentation Theory and Rhetoric, 19(1), 97–102. https://philpapers.org/archive/OBAPFQ.pdf
37. Maddox, B., Bayliss, A. P., Fleming, P., Engelhardt, P. E., Edwards, S. G., & Borgonovi, F. (2018). Observing response processes with Eye Tracking in international large-scale assessments: evidence from the OECD PIAAC assessment. European Journal of Psychology of Education, 33(3), 543–558. https://doi.org/10.1007/s10212-018-0380-2
38. Maia, C. A. A.S, Negreiros, F. D. S., & Moreira, T. M. M. (2023). Evaluative instruments of patients' knowledge about systematic arterial hypertension and diabetes mellitus: a scoping review. Nurse Education in Practice, 103838. https://doi.org/10.1016/j.nepr.2023.103838
39. Man, K., Harring, J. R., & Zhan, P. (2022). Bridging Models of Biometric and Psychometric Assessment: A Three-Way Joint Modeling Approach of Item Responses, Response Times, and Gaze Fixation Counts. Applied Psychological Measurement, 46(5), 361-381. https://doi.org/10.1177/01466216221089344
40. Milte, R., Crocker, M., Lay, K., Ratcliffe, J., Mulhern, B., Norman, R., Viney, R., & Khadka, J. (2023). Feasibility of self-reported health related quality of life assessment with older people in residential care: insights from the application of Eye Tracking technology. Quality of Life Research. https://doi.org/10.1007/s11136-023-03488-w
41. Monteiro, S. D., Rodas, C. M., & Vidotti, S. A. B. G. (2020). A busca e o Eye Tracking: um olhar semiótico sobre o knowledge graph. Em Questão, 26(1), 304–326. https://doi.org/10.19132/1808-5245261.304-326
42. Neuert, C. E. (2019). How Effective Are Eye-Tracking Data in Identifying Problematic Questions? Social Science Computer Review, 38(6), 793–802. https://doi.org/10.1177/0894439319834289
43. Padilla, J. L., & Benítez, I. (2014). Validity evidence based on response processes. Psicothema, 26(1), 136–144. https://doi.org/10.7334/psicothema2013.259
44. Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., ... Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, n71. https://doi.org/10.1136/bmj.n71
45. Pauszek, J. R. (2022). An Introduction to Eye Tracking in Human Factors Healthcare Research and Medical Device Testing. Human Factors in Healthcare, 100031. https://doi.org/10.1016/j.hfh.2022.100031
46. Peters, M., Godfrey, C., McInerney, P., Munn, Z., Trico, A., & Khalil, H. (2020). Chapter 11: Scoping Reviews. In JBI Manual for Evidence Synthesis. JBI. https://doi.org/10.46658/jbimes-20-12
47. Ratcliffe, J., Lay, K., Crocker, M., Engel, L., Milte, R., Hutchinson, C., Khadka, J., Whitehurst, D. G. T., Mulhern, B., Viney, R., & Norman, R. (2023). Unravelling the Self-Report Versus Proxy-Report Conundrum for Older Aged Care Residents: Findings from a Mixed-Methods Study. The Patient - Patient-Centered Outcomes Research. https://doi.org/10.1007/s40271-023-00655-6
48. Reppold, C.T., Zanini, DS., & Noronha, A. P. (2019). What is psychological assessment? In M.N. Baptista et al. (Eds), Compendium of psychological Assesmment. Petropolis, RJ: Vozes. ISBN 978-85-326-6360-3. https://abrir.link/Y5t4J
49. Requena, F. C. G. (2022). Visual analysis of reading patterns in Portuguese texts through Eye Tracking (Master's dissertation, FEI University Center, São Bernardo do Campo). Available at: https://repositorio.fei.edu.br/items/61e221d4-a51f-4747-8281-b0f4de3b8a04
50. Rivu, R., Abdrabou, Y., Abdelrahman, Y., Pfeuffer, K., Kern, D., Neuert, C., Buschek, D., & Alt, F. (2021). Did you Understand this? In ETRA '21: 2021 Symposium on Eye Tracking Research and Applications. ACM. https://doi.org/10.1145/3448018.3458018
51. Rodrigues, E. S. (2021). Analysis of eye-tracking data in game evaluation: A case study (Master's dissertation). Federal University of Campina Grande. Available from http://dspace.sti.ufcg.edu.br:8080/jspui/handle/riufcg/21364
52. Tontini, J. (2021). Information processing and Eye Tracking in pre-test research (Master's dissertation, Postgraduate Program in Administration, Federal University of Santa Maria). Santa Maria, RS: Universidade Federal de Santa Maria. Retrieved from https://repositorio.ufsm.br/handle/1/25677
53. Wang, K., Barr, C., Norman, R., George, S., Whitehead, C., & Ratcliffe, J. (2020). Using Eye-Tracking Technology with Older People in Memory Clinics to Investigate the Impact of Mild Cognitive Impairment on Choices for EQ-5D-5L Health States Preferences. Applied Health Economics and Health Policy, 19(1), 111–121. https://doi.org/10.1007/s40258-020-00588-3
54. Yaneva, V., Clauser, B. E., Morales, A., & Paniagua, M. (2021). Using Eye‐Tracking Data as Part of the Validity Argument for Multiple‐Choice Questions: A Demonstration. Journal of Educational Measurement, 58(4), 515–537. https://doi.org/10.1111/jedm.12304
55. Yaneva, V., Clauser, B. E., Morales, A., & Paniagua, M. (2022). Assessing the validity of test scores using response process data from an eye-tracking study: a new approach. Advances in Health Sciences Education. https://doi.org/10.1007/s10459-022-10107-9