Applications of Optical Brain Imaging Methods in Aviation Neuroergonomics

Abstract

The development of objective methods that enable monitoring of the cognitive status of pilots, UAV operators, and air traffic controllers is critically important in aviation for improving flight safety, optimizing pilot/operator training and developing innovative man-machine interfaces. Functional near-infrared spectroscopy (fNIRS) optical brain imaging technology offers significant advantages for this purpose by providing portable, rugged sensors that can be employed in the field to monitor neurophysiological markers during flight operations. This article reviews studies that employ fNIRS technology for cognitive workload assessment, operator interface evaluation, and g-loc/hypoxia prediction in aviation to document the potential of neurophysiological measurement modalities like fNIRS for aviation medicine and ergonomics.

Keywords

• Neuroergonomics
• Optical Brain Imaging
• Functional Near-infrared Spectroscopy
• Aviation Medicine
• Aviation Psychology

Havacılık Nöroergonomisinde Optik Beyin Görüntüleme Uygulamaları

Abstract

Pilotların, insansız hava aracı operatörlerinin, hava trafik kontrolörlerinin eğitim ve uçuş faaliyetleri sırasında bilişsel durumlarının takibini sağlayacak nesnel yöntemlerin geliştirilmesi uçuş emniyetinin sağlanması, eğitim süreçlerinin optimizasyonu ve yenilikçi insan-makine arayüzlerinin tasarımı bakımından kritik önem taşımaktadır. İşlevsel Yakın-Kızılötesi Tayfölçümü (functional near infrared spectroscopy – fNIRS) optik beyin görüntüleme teknolojisi gibi saha kullanımına uygun, portatif ve güvenilir nörofizyolojik ölçüm yöntemleri bu ihtiyaçlara yönelik bazı önemli avantajlar sunmaktadır. Bu derlemede fNIRS teknolojisinin dayandığı bilimsel temeller ve bu teknolojiyle gerçekleştirilmiş pilot/operatör bilişsel işyükü takibi, kontrol arayüzü değerlendirmesi, g-loc/hipoksi kestirimi gibi öncü havacılık uygulamalarından örnekler sunularak fNIRS yönteminin havacılık tıbbı ve ergonomisi alanları için sunduğu imkanların özetlenmesi amaçlanmıştır.

Keywords

• Nöroergonomi
• Optik Beyin Görüntüleme
• İşlevsel Kızıl-ötesi Tayfölçümü
• Havacılık Tıbbı
• Havacılık Psikolojisi

References

1. Ayaz, H., Onaral, B., Izzetoglu, K., Shewokis, P. A., McKendrick, R., & Parasuraman, R. (2013). Continuous monitoring of brain dynamics with functional near infrared spectroscopy as a tool for neuroergonomic research: Empirical examples and a technological development. Frontiers in Human Neuroscience, 7, 1-13.
2. Ayaz, H., Çakir, M. P., İzzetoğlu, K., Curtin, A., Shewokis, P. A., Bunce, S. C., & Onaral, B. (2012, March). Monitoring expertise development during simulated UAV piloting tasks using optical brain imaging. In Proceedings of 2012 IEEE aerospace conference (pp. 1-11). IEEE.
3. Ayaz, H., Shewokis, P. A., Bunce, S., Izzetoglu, K., Willems, B., & Onaral, B. (2012). Optical brain monitoring for operator training and mental workload assessment. Neuroimage, 59(1), 36-47.
4. Ayaz, H., Bunce, S., Shewokis, P., Izzetoglu, K., Willems, B., & Onaral, B. (2012). Using Brain Activity to Predict Task Performance and Operator Efficiency. In H. Zhang, A. Hussain, D. Liu & Z. Wang (Eds.), Advances in Brain Inspired Cognitive Systems (Vol. 7366, pp. 147-155). Berlin: Springer.
5. Ayaz, H., Shewokis, P. A., Curtin, A., Izzetoglu, M., Izzetoglu, K., & Onaral, B. (2011). Using MazeSuite and Functional Near Infrared Spectroscopy to Study Learning in Spatial Navigation. Journal of Visual Experiments (56),e3443.
6. Ayaz, H., Willems, B., Bunce, B., Shewokis, P. A., Izzetoglu, K., Hah, S., Onaral, B. (2010). Cognitive Workload Assessment of Air Traffic Controllers Using Optical Brain Imaging Sensors. In T. Marek, W. Karwowski & V. Rice (Eds.), Advances in Understanding Human Performance: Neuroergonomics, Human Factors Design, and Special Populations (pp. 21-31): CRC Press.
7. Ayaz, H., Izzetoglu, M., Shewokis, P. A., & Onaral, B. (2010, August). Sliding-window motion artifact rejection for functional near-infrared spectroscopy. In Proceedings of 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology (pp. 6567-6570). IEEE.
8. Ayaz, H., & Dehais, F. (Eds.). (2019). Neuroergonomics: The brain at work and in everyday life. Academic Press.

9. Barker, J. W., Aarabi, A., & Huppert, T. J. (2013). Autoregressive model based algorithm for correcting motion and serially correlated errors in fNIRS. Biomedical Optics Express, 4(8), 1366-1379.
10. Benni, P. B., Li, J. K., Chen, B., Cammarota, J., & Amory, D. W. (2003). NIRS monitoring of pilots subjected to+ Gz acceleration and G-induced loss of consciousness (G-LOC). In Oxygen Transport to Tissue XXIV (pp. 371-379). New York: Springer.
11. Borghini, G., Isabella, R., Vecchiato, G., Toppi, J., Astolfi, L., Caltagirone, C., & Babiloni, F. (2011). Brainshield: HREEG study of perceived pilot mental workload. Italian Journal of Aerospace Medicine, 5, 34-47.
12. Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., & Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience & Biobehavioral Reviews, 44, 58-75.
13. Bunce, S., Izzetoglu, K., Ayaz, H., Shewokis, P., Izzetoglu, M., Pourrezaei, K., & Onaral, B. (2011). Implementation of fNIRS for Monitoring Levels of Expertise and Mental Workload. In D. Schmorrow & C. Fidopiastis (Eds.), Foundations of Augmented Cognition. Directing the Future of Adaptive Systems (Vol. 6780, pp. 13-22). Berlin: Springer.
14. Buxton, R. B., Wong, E. C., & Frank, L. R. (1998). Dynamics of blood flow and oxygenation changes during brain activation: the balloon model. Magnetic Resonance in Medicine, 39(6), 855-864.
15. Buxton, R. B., Uludağ, K., Dubowitz, D. J., & Liu, T. T. (2004). Modeling the hemodynamic response to brain activation. Neuroimage, 23, S220-S233.
16. Çakır M.P., Vural M., Koç S.Ö., Toktaş A. (2016) Real-Time Monitoring of Cognitive Workload of Airline Pilots in a Flight Simulator with fNIR Optical Brain Imaging Technology. In: Schmorrow D., Fidopiastis C. (eds) Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience. AC 2016. Lecture Notes in Computer Science, vol 9743. New York: Springer.
17. Chance, B. (1998). Near-infrared images using continuous, phase-modulated, and pulsed light with quantitation of blood and blood oxygenation. Annals of New York Academy of Sciences, 838, 29-45.
18. Chance, B. (1991). Optical method. Annual Review of Biophysics and Biophysical Chemistry, 20(1), 1-30.
19. Chance, B., Anday, E., Nioka, S., Zhou, S., Hong, L., Worden, K., . . . Thomas, R. (1998). A novel method for fast imaging of brain function, non-invasively, with light. Optics Express, 2(10), 411-423.
20. Dahlstrom, N., Dekker, S., Van Winsen, R., & Nyce, J. (2009). Fidelity and validity of simulator training. Theoretical Issues in Ergonomics Science, 10(4), 305-314.
21. Davranche, K., Casini, L., Arnal, P. J., Rupp, T., Perrey, S., & Verges, S. (2016). Cognitive functions and cerebral oxygenation changes during acute and prolonged hypoxic exposure. Physiology & Behavior, 164, 189-197.
22. De Rivecourt, M., Kuperus, M. N., Post, W. J., & Mulder, L. J. M. (2008). Cardiovascular and eye activity measures as indices for momentary changes in mental effort during simulated flight. Ergonomics, 51(9), 1295-1319.
23. Ferrari, M., & Quaresima, V. (2012). A brief review on the history of human functional near-infrared spectroscopy (fNIRS) development and fields of application. Neuroimage, 63(2), 921-935.
24. Frijia, E. M., Billing, A., Lloyd-Fox, S., Rosas, E. V., Collins-Jones, L., Crespo-Llado, M. M., ... & Cooper, R. J. (2021). Functional imaging of the developing brain with wearable high-density diffuse optical tomography: A new benchmark for infant neuroimaging outside the scanner environment. Neuroimage, 225, 117490.
25. Haarmann, A., Boucsein, W., & Schaefer, F. (2009). Combining electrodermal responses and cardiovascular measures for probing adaptive automation during simulated flight. Applied Ergonomics, 40(6), 1026-1040. Haier, R. J., Karama, S., Leyba, L., & Jung, R. E. (2009). MRI assessment of cortical thickness and functional activity changes in adolescent girls following three months of practice on a visual-spatial task. BMC Research Notes, 2(1), 1-7.
26. Hays, R. T., Jacobs, J. W., Prince, C., & Salas, E. (1992). Requirements for future research in flight simulation training: Guidance based on a meta-analytic review. The International Journal of Aviation Psychology, 2(2), 143-158.
27. Jobsis, F. F. (1977). Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science, 198(4323), 1264-1267.
28. Harrison, J., Izzetoglu, K., Ayaz, H., Willems, B., Hah, S., Ahlstrom, U., Woo, H., Shewokis, P., Bunce, S.C., Onaral, B. (2014). Cognitive Workload and Learning Assessment During the Implementation of a Next-Generation Air Traffic Control Technology Using Functional Near-Infrared Spectroscopy. IEEE Transactions in Human-Machine Systems 44(4), 429-440.
29. Harrison, J., Izzetoglu, K., Ayaz, H., Willems, B., Hah, S., Woo, H., . . . Onaral, B. (2013). Human Performance Assessment Study in Aviation Using Functional Near Infrared Spectroscopy. In D. Schmorrow & C. Fidopiastis (Eds.), Foundations of Augmented Cognition (Vol. 8027, pp. 433-442), Berlin: Springer.
30. Heeger, D. J., & Ress, D. (2002). What does fMRI tell us about neuronal activity? Nature Reviews Neuroscience, 3(2), 142-151.
31. Hoshi, Y., & Tamura, M. (1997). Near-infrared optical detection of sequential brain activation in the prefrontal cortex during mental tasks. Neuroimage, 5(4), 292-297.
32. Hoshi, Y., & Tamura, M. (1993). Detection of dynamic changes in cerebral oxygenation coupled to neuronal function during mental work in man. Neuroscience Letters, 150(1), 5.
33. Iadecola, C. (2017). The neurovascular unit coming of age: a journey through neurovascular coupling in health and disease. Neuron, 96(1), 17-42.
34. ICAO (2014). Safet Report. International Civil Aviation Organization. http://www.icao.int/safety/Documents/ICAO_ 2014%20Safety%20Report_final_02042014_web.pdf
35. Izzetoglu, K., Ayaz, H., Menda, J., Izzetoglu, M., Merzagora, A., Shewokis, P., Onaral, B. (2011). Applications of Functional Near Infrared Imaging: Case Study on UAV Ground Controller. In D. Schmorrow & C. Fidopiastis (Eds.), Foundations of Augmented Cognition. Directing the Future of Adaptive Systems (Vol. 6780, pp. 608-617). New York: Springer.
36. Izzetoglu, K., Bunce, S., Onaral, B., Pourrezaei, K., & Chance, B. (2004). Functional Optical Brain Imaging Using Near-Infrared During Cognitive Tasks. International Journal of Human-Computer Interaction, 17(2), 211-227.
37. Kikukawa, A., Kobayashi, A., & Miyamoto, Y. (2008). Monitoring of pre-frontal oxygen status in helicopter pilots using near-infrared spectrophotometers. Dynamic Medicine, 7(1), doi:10.1186/1476-5918-7-10.
38. Kobayashi, A., & Miyamoto, Y. (2000). In-flight cerebral oxygen status: continuous monitoring by near-infrared spectroscopy. Aviation Space and Environmental Medicine, 71(2), 177-183.
39. Kurihara, K., Kikukawa, A., Kobayashi, A., & Nakadate, T. (2007). Frontal cortical oxygenation changes during gravity-induced loss of consciousness in humans: a near-infrared spatially resolved spectroscopic study. Journal of Applied Physiology, 103(4), 1326-1331.
40. Mannheimer, P. D. (2007). The light–tissue interaction of pulse oximetry. Anesthesia & Analgesia, 105(6), S10-S17.
41. Menda, J., Hing, J. T., Ayaz, H., Shewokis, P. A., Izzetoglu, K., Onaral, B., & Oh, P. (2011). Optical brain imaging to enhance UAV operator training, evaluation, and interface development. Journal of intelligent & robotic systems, 61(1-4), 423-443.
42. Molavi, B., & Dumont, G. A. (2012). Wavelet-based motion artifact removal for functional near-infrared spectroscopy. Physiological Measurement, 33(2), 259.
43. Parasuraman, R., & Wickens, C. D. (2008). Humans: Still vital after all these years of automation. Human factors, 50(3), 511-520.
44. Pifferi, A., Contini, D., Dalla Mora, A., Farina, A., Spinelli, L., & Torricelli, A. (2016). New frontiers in time-domain diffuse optics, a review. Journal of Biomedical Optics, 21(9), 091310.
45. Pinti, P., Aichelburg, C., Gilbert, S., Hamilton, A., Hirsch, J., Burgess, P., & Tachtsidis, I. (2018). A review on the use of wearable functional near‐infrared spectroscopy in naturalistic environments. Japanese Psychological Research, 60(4), 347-373.
46. Quaresima, V., & Ferrari, M. (2019). Functional near-infrared spectroscopy (fNIRS) for assessing cerebral cortex function during human behavior in natural/social situations: a concise review. Organizational Research Methods, 22(1), 46-68.
47. Ryoo, H. C., Sun, H. H., Shender, B. S., & Hrebien, L. (2004). Consciousness monitoring using near-infrared spectroscopy (NIRS) during high+ Gz exposures. Medical Engineering & Physics, 26(9), 745-753. Salas, E., Bowers, C. A., & Rhodenizer, L. (1998). It is not how much you have but how you use it: Toward a rational use of simulation to support aviation training. The International Journal of Aviation Psychology, 8(3), 197-208.
48. Scholl, C. A., Chi, Y. M., Elconin, M., Gray, W. R., Chevillet, M. A., & Pohlmeyer, E. A. (2016). Classification of pilot-induced oscillations during in-flight piloting exercises using dry EEG sensor recordings. In 2016 IEEE 38th Annual International Conference of the Engineering in Medicine and Biology Society (EMBC) (pp. 4467-4470). IEEE.
49. Shewokis, P. A., Ayaz, H., Izzetoglu, M., Bunce, S., Gentili, R. J., Sela, I. & Onaral, B. (2011). Brain in the loop: assessing learning using fNIR in cognitive and motor tasks. In Schmorrow D.D., Fidopiastis C.M. (eds) Foundations of Augmented Cognition. Directing the Future of Adaptive Systems. FAC 2011. Lecture Notes in Computer Science, vol 6780 (pp. 240-249). Berlin: Springer.
50. Strangman, G., Boas, D. A., & Sutton, J. P. (2002). Non-invasive neuroimaging using near-infrared light. Biological Psychiatry, 52(7), 679-693.
51. Sweeney, K. T., Ayaz, H., Ward, T. E., Izzetoglu, M., McLoone, S. F., & Onaral, B. (2012). A methodology for validating artifact removal techniques for physiological signals. IEEE Transactions on Information Technology in Biomedicine, 16(5), 918–926.
52. van Dijk, H., van de Merwe, K., & Zon, R. (2011). A Coherent Impression of the Pilots' Situation Awareness: Studying Relevant Human Factors Tools. International Journal of Aviation Psychology, 21(4), 343-356.
53. Wickens, C. D. (2008). Multiple resources and mental workload. Human Factors, 50(3), 449-455.
54. Wickens, C. D. (2014). Effort in human factors performance and decision making. Human Factors, 56(8), 1329-1336.
55. Villringer, A., & Chance, B. (1997). Non-invasive optical spectroscopy and imaging of human brain function. Trends in Neurosciences, 20(10), 435-442.
56. Wilson, G. F. (2002). An analysis of mental workload in pilots during flight using multiple psychophysiological measures. International Journal of Aviation Psychology, 12(1), 3-18.
57. Wilson, G. F., Caldwell, J. A., & Russell, C. A. (2007). Performance and psychophysiological measures of fatigue effects on aviation related tasks of varying difficulty. International journal of aviation psychology, 17(2), 219-247.
58. Wilson, G. F., & Russell, C. A. (2007). Performance enhancement in an uninhabited air vehicle task using psychophysiologically determined adaptive aiding. Human Factors, 49(6), 1005-1018.
59. Yamada, Y., Suzuki, H., & Yamashita, Y. (2019). Time-domain near-infrared spectroscopy and imaging: A review. Applied Sciences, 9(6), 1127.
60. Yeh, Y. Y., & Wickens, C. D. (1988). Dissociation of performance and subjective measures of workload. Human Factors, 30(1), 111-120.