Innovative Approach in Action Observation Training: Full Immersive Virtual Reality

Abstract

Action observation training is a neurophysiological approach based on the mirror neuron system that facilitates cortical reorganization. The method involves the systematic observation of goal-directed actions, followed by their physical execution. During action observation training, mirror neuron responses are modulated by multiple factors: action-related factors, actor-related factors, observer-related factors, the relationship between the actor and observer, and contextual factors. To ensure effective implementation of action observation training, it is important to consider modulating parameters such as motivation, degree of realism, perspective, awareness, attention, and postural alignment. In this regard, virtual reality technologies are increasingly utilized in action observation training due to their various advantages. Based on the level of immersion, virtual reality systems are generally classified into three categories: non-immersive, semi-immersive, and fully immersive. Recently, fully immersive virtual reality has been integrated into action observation training to enhance interaction with diverse environments, increase the degree of realism, and enable a first-person perspective. Although the research in this area remains limited, existing studies have demonstrated the positive effects of combining action observation training with fully immersive virtual reality. This review aims to provide an overview of the factors that modulate action observation training, with particular emphasis on its integration with fully immersive virtual reality.

Keywords

• mirror neuron
• action observation training
• virtual reality
• head-mounted display

Eylem Gözlem Eğitiminde Yenilikçi Yaklaşım: Tam Daldırmalı Sanal Gerçeklik

Abstract

Eylem gözlem eğitimi, ayna nöron sistemine dayanan, kortikal reorganizasyonu destekleyen nörofizyolojik bir yaklaşımdır. Bu yöntem, eylemlerin sistematik olarak gözlemlenmesini ve ardından bu eylemlerin uygulanmasını esas alır. Eylem gözlem eğitimi sırasında ayna nöron yanıtları eylemle ilgili faktörler, aktörle ilgili faktörler, gözlemci ile ilgili faktörler, aktör ve gözlemci arasındaki ilişkiyi içeren faktörler ve bağlam faktörü olmak üzere birden fazla faktör tarafından modüle edilmektedir. Motivasyon, gerçeklik derecesi, perspektif, farkındalık, dikkat, postüral uyum gibi parametrelerin etkili bir eylem gözlem eğitimi gerçekleştirilmesi için dikkate alınması önemlidir. Bu bakımdan çeşitli avantajlarıyla eylem gözlem eğitiminde sanal gerçeklik uygulamaları kullanılmaktadır. Sanal gerçeklik uygulamaları, daldırma durumlarına göre daldırma olmayan (non-immersive), kısmi daldırmalı (semi-immersive) ve tam daldırmalı (full immersive) olmak üzere üç genel kategoride gruplandırılmaktadır. Son zamanlarda tam daldırmalı sanal gerçeklik uygulamaları bireyin farklı ortamlarla etkileşim içinde olması, gerçeklik algısının artırılması ve birinci kişi bakış açısının sağlanması için eylem gözlem eğitiminde kullanılmaya başlanmıştır. Bununla ilgili sınırlı sayıda çalışmalar olsa da çalışma sonuçlarında tam daldırmalı sanal gerçeklikle uygulanan eylem gözlem eğitiminin olumlu etkileri ortaya koyulmuştur. Bu bağlamda, bu derleme eylem gözlem eğitimini modüle eden faktörler ve tam daldırmalı sanal gerçeklikle uygulanan eylem gözlem eğitimi üzerine genel bir bakış sunmaktadır.

Keywords

• ayna nöron
• eylem gözlem eğitimi
• sanal gerçeklik
• sanal gerçeklik başlığı

References

1. Ryan D, Fullen B, Rio E, Segurado R, Stokes D, O’Sullivan C. Effect of action observation therapy in the rehabilitation of neurologic and musculoskeletal conditions: a systematic review. Archives of rehabilitation research and clinical translation. 2021;3(1):100106.
2. Gazzola V, Keysers C. The observation and execution of actions share motor and somatosensory voxels in all tested subjects: single-subject analyses of unsmoothed fMRI data. Cerebral cortex. 2009;19(6):1239-55.
3. Molenberghs P, Cunnington R, Mattingley JB. Brain regions with mirror properties: a meta-analysis of 125 human fMRI studies. Neuroscience & biobehavioral reviews. 2012;36(1):341-9.
4. Van Overwalle F, Baetens K. Understanding others' actions and goals by mirror and mentalizing systems: a meta-analysis. Neuroimage. 2009;48(3):564-84.
5. Sallés L, Gironès X, Martín-Casas P, Lafuente JV. A neurocognitive approach to recovery of movement following stroke. Physical Therapy Reviews. 2015;20(5-6):283-9.
6. Zhu M-H, Wang J, Gu X-D, Shi M-F, Zeng M, Wang C-Y, et al. Effect of action observation therapy on daily activities and motor recovery in stroke patients. International Journal of Nursing Sciences. 2015;2(3):279-82.
7. Buccino G. Action observation treatment: a novel tool in neurorehabilitation. Philosophical Transactions of the Royal Society B: Biological Sciences. 2014;369(1644):20130185.
8. Sarasso E, Gemma M, Agosta F, Filippi M, Gatti R. Action observation training to improve motor function recovery: a systematic review. Archives of physiotherapy. 2015;5:1-12.

9. Vogt S, Thomaschke R. From visuo-motor interactions to imitation learning: behavioural and brain imaging studies. Journal of sports sciences. 2007;25(5):497-517.
10. Borges LR, Fernandes AB, Dos Passos JO, Rego IAO, Campos TF. Action observation for upper limb rehabilitation after stroke. Cochrane Database of Systematic Reviews. 2022(8).
11. Kwok JYY, Smith R, Chan LML, Lam LCC, Fong DYT, Choi EPH, et al. Managing freezing of gait in Parkinson’s disease: a systematic review and network meta-analysis. Journal of Neurology. 2022;269(6):3310-24.
12. Te Velde A, Morgan C, Finch-Edmondson M, McNamara L, McNamara M, Paton MCB, et al. Neurodevelopmental therapy for cerebral palsy: a meta-analysis. Pediatrics. 2022;149(6):e2021055061.
13. Kemmerer D. What modulates the mirror neuron system during action observation?: Multiple factors involving the action, the actor, the observer, the relationship between actor and observer, and the context. Progress in Neurobiology. 2021;205:102128.
14. Jola C, Grosbras M-H. In the here and now: Enhanced motor corticospinal excitability in novices when watching live compared to video recorded dance. Cognitive neuroscience. 2013;4(2):90-8.
15. Prinsen J, Alaerts K. Eye contact enhances interpersonal motor resonance: Comparing video stimuli to a live two-person action context. Social Cognitive and Affective Neuroscience. 2019;14(9):967-76.
16. Ruysschaert L, Warreyn P, Wiersema JR, Metin B, Roeyers H. Neural mirroring during the observation of live and video actions in infants. Clinical Neurophysiology. 2013;124(9):1765-70.
17. Sollfrank T, Hart D, Goodsell R, Foster J, Tan T. 3D visualization of movements can amplify motor cortex activation during subsequent motor imagery. Frontiers in human Neuroscience. 2015;9:463.
18. Ferri S, Pauwels K, Rizzolatti G, Orban G. Stereoscopically observing manipulative actions. Cerebral cortex. 2016;26(8):3591-610.
19. Jastorff J, Abdollahi RO, Fasano F, Orban GA. Seeing biological actions in 3 D: An f MRI study. Human Brain Mapping. 2016;37(1):203-19.
20. Buccino G, Lui F, Canessa N, Patteri I, Lagravinese G, Benuzzi F, et al. Neural circuits involved in the recognition of actions performed by nonconspecifics: An fMRI study. Journal of cognitive neuroscience. 2004;16(1):114-26.
21. Miura N, Sugiura M, Takahashi M, Sassa Y, Miyamoto A, Sato S, et al. Effect of motion smoothness on brain activity while observing a dance: An fMRI study using a humanoid robot. Social neuroscience. 2010;5(1):40-58.
22. Tai YF, Scherfler C, Brooks DJ, Sawamoto N, Castiello U. The human premotor cortex is' mirror'only for biological actions. Current biology. 2004;14(2):117-20.
23. Fox NA, Bakermans-Kranenburg MJ, Yoo KH, Bowman LC, Cannon EN, Vanderwert RE, et al. Assessing human mirror activity with EEG mu rhythm: A meta-analysis. Psychological bulletin. 2016;142(3):291.
24. Hogeveen J, Obhi SS. Social interaction enhances motor resonance for observed human actions. Journal of Neuroscience. 2012;32(17):5984-9.
25. Nagai H, Tanaka T. Action observation of own hand movement enhances event-related desynchronization. IEEE transactions on neural systems and rehabilitation engineering. 2019;27(7):1407-15.
26. Angelini M, Fabbri-Destro M, Lopomo NF, Gobbo M, Rizzolatti G, Avanzini P. Perspective-dependent reactivity of sensorimotor mu rhythm in alpha and beta ranges during action observation: an EEG study. Scientific reports. 2018;8(1):12429.
27. Ge S, Liu H, Lin P, Gao J, Xiao C, Li Z. Neural basis of action observation and understanding from first-and third-person perspectives: an fMRI study. Frontiers in behavioral neuroscience. 2018;12:283.
28. Maeda F, Kleiner-Fisman G, Pascual-Leone A. Motor facilitation while observing hand actions: specificity of the effect and role of observer's orientation. Journal of neurophysiology. 2002;87(3):1329-35.
29. Bunday KL, Lemon RN, Kilner JM, Davare M, Orban GA. Grasp-specific motor resonance is influenced by the visibility of the observed actor. Cortex. 2016;84:43-54.
30. Heimann K, Umiltà MA, Guerra M, Gallese V. Moving mirrors: a high-density EEG study investigating the effect of camera movements on motor cortex activation during action observation. Journal of cognitive neuroscience. 2014;26(9):2087-101.
31. Alaerts K, Heremans E, Swinnen SP, Wenderoth N. How are observed actions mapped to the observer’s motor system? Influence of posture and perspective. Neuropsychologia. 2009;47(2):415-22.
32. Urgesi C, Moro V, Candidi M, Aglioti SM. Mapping implied body actions in the human motor system. Journal of Neuroscience. 2006;26(30):7942-9.
33. Liew SL, Han S, Aziz‐Zadeh L. Familiarity modulates mirror neuron and mentalizing regions during intention understanding. Human brain mapping. 2011;32(11):1986-97.
34. Riach M, Holmes PS, Franklin ZC, Wright DJ. Observation of an action with a congruent contextual background facilitates corticospinal excitability: A combined TMS and eye-tracking experiment. Neuropsychologia. 2018;119:157-64.
35. Amoruso L, Finisguerra A, Urgesi C. Tracking the time course of top-down contextual effects on motor responses during action comprehension. Journal of neuroscience. 2016;36(46):11590-600.
36. Walsh KR, Pawlowski SD. Virtual reality: A technology in need of IS research. Communications of the Association for Information Systems. 2002;8(1):20.
37. Sanchez-Vives MV, Slater M. From presence to consciousness through virtual reality. Nature reviews neuroscience. 2005;6(4):332-9.
38. Wohlgenannt I, Simons A, Stieglitz S. Virtual reality. Business & Information Systems Engineering. 2020;62:455-61.
39. Slater M, Linakis V, Usoh M, Kooper R, editors. Immersion, presence and performance in virtual environments: An experiment with tri-dimensional chess. Proceedings of the ACM symposium on virtual reality software and technology; 1996.
40. Rose T, Nam CS, Chen KB. Immersion of virtual reality for rehabilitation-Review. Applied ergonomics. 2018;69:153-61.
41. Lum HC, Greatbatch R, Waldfogle G, Benedict J, editors. How immersion, presence, emotion, & workload differ in virtual reality and traditional game mediums. Proceedings of the Human Factors and Ergonomics Society Annual Meeting; 2018: SAGE Publications Sage CA: Los Angeles, CA.
42. Yao S, Kim G, editors. The effects of immersion in a virtual reality game: Presence and physical activity. HCI in Games: First International Conference, HCI-Games 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26–31, 2019, Proceedings 21; 2019: Springer.
43. Burdea GC. Virtual rehabilitation–benefits and challenges. Methods of information in medicine. 2003;42(05):519-23.
44. Lange B, Koenig S, Chang C-Y, McConnell E, Suma E, Bolas M, et al. Designing informed game-based rehabilitation tasks leveraging advances in virtual reality. Disability and rehabilitation. 2012;34(22):1863-70.
45. Paltamaa J, Sjögren T, Peurala S, Heinonen A. Effects ofphysiotherapy interventions on balance in multiple sclerosis: asystematic review and meta-analysis of randomized controlled trials.Journal of rehabilitation medicine. 2012;44(10).
46. Keshner EA, Fung J. The quest to apply VR technology torehabilitation: tribulations and treasures. Journal of VestibularResearch. 2017;27(1):1-5.
47. Levin MF, Weiss PL, Keshner EA. Emergence of virtual reality as a toolfor upper limb rehabilitation: incorporation of motor control andmotor learning principles. Physical therapy. 2015;95(3):415-25.
48. Merians AS, Jack D, Boian R, Tremaine M, Burdea GC, Adamovich SV,et al. Virtual reality–augmented rehabilitation for patients followingstroke. Physical therapy. 2002;82(9):898-915.
49. Calabrò RS, Russo M, Naro A, De Luca R, Leo A, Tomasello P, et al.Robotic gait training in multiple sclerosis rehabilitation: Can virtualreality make the difference? Findings from a randomized controlledtrial. Journal of the neurological sciences. 2017;377:25-30.
50. 50.Bertoni R, Mattos FGM, Porta M, Arippa F, Cocco E, Pau M, et al.Effects of immersive virtual reality on upper limb function in subjectswith multiple sclerosis: A cross-over study. Multiple Sclerosis andRelated Disorders. 2022;65:104004.
51. Slater M. Immersion and the illusion of presence in virtual reality.British journal of psychology. 2018;109(3):431-3.
52. Slater M, Pérez Marcos D, Ehrsson H, Sanchez-Vives MV. Towards adigital body: the virtual arm illusion. Frontiers in humanneuroscience. 2008;2:181.
53. Banakou D, Slater M. Body ownership causes illusory self-attributionof speaking and influences subsequent real speaking. Proceedingsof the National Academy of Sciences. 2014;111(49):17678-83.
54. Kilteni K, Normand J-M, Sanchez-Vives MV, Slater M. Extendingbody space in immersive virtual reality: a very long arm illusion. PloSone. 2012;7(7):e40867.
55. Argelaguet F, Hoyet L, Trico M, Lécuyer A, editors. The role ofinteraction in virtual embodiment: Effects of the virtual handrepresentation. 2016 IEEE virtual reality (VR); 2016: IEEE.
56. Nuara A, Fabbri-Destro M, Scalona E, Lenzi SE, Rizzolatti G, AvanziniP. Telerehabilitation in response to constrained physical distance: anopportunity to rethink neurorehabilitative routines. Journal ofneurology. 2022;269(2):627-38.
57. Amoruso L, Urgesi C. Contextual modulation of motor resonanceduring the observation of everyday actions. NeuroImage.2016;134:74-84.
58. Laver KE, Lange B, George S, Deutsch JE, Saposnik G, Crotty M. Virtual reality for stroke rehabilitation. Cochrane database of systematic reviews. 2017(11).
59. Burin-Chu S, Baillet H, Leconte P, Lejeune L, Thouvarecq R, Benguigui N. Effectiveness of virtual reality interventions of the upper limb in children and young adults with cerebral palsy: A systematic review with meta-analysis. Clinical Rehabilitation. 2024;38(1):15-33.
60. Choi JW, Kim BH, Huh S, Jo S. Observing actions through immersive virtual reality enhances motor imagery training. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2020;28(7):1614-22.
61. Yoshimura M, Kurumadani H, Hirata J, Osaka H, Senoo K, Date S, et al. Virtual reality-based action observation facilitates the acquisition of body-powered prosthetic control skills. Journal of NeuroEngineering and Rehabilitation. 2020;17:1-12.
62. Emedoli D, Arosio M, Tettamanti A, Iannaccone S. Virtual reality augmented feedback rehabilitation associated to action observation therapy in buccofacial apraxia: Case Report. Clinical Medicine Insights: Case Reports. 2021;14:1179547621994579.
63. Adamo P, De Leo D, Fava T, Binda S, Avanzini P, Fabbri-Destro M, et al. Effects of immersive virtual reality-based action observation training integrated with focal muscle vibration on upper extremity in a patient with chronic disability following stroke: a single-subject study. Physiotherapy Theory and Practice. 2025:1-9.
64. Lakshminarayanan K, Shah R, Daulat SR, Moodley V, Yao Y, MadathilD.The effect of combining action observation in virtual reality with kinesthetic motor imagery on cortical activity. Frontiers in Neuroscience. 2023;17:1201865.
65. Adamo P, Longhi G, Temporiti F, Marino G, Scalona E, Fabbri-Destro M, et al. Effects of action observation plus motor imagery administered by immersive virtual reality on hand dexterity in healthy subjects. Bioengineering. 2024;11(4):398.
66. Pinkl J, Cohen M, editors. Design of a VR action observation tool for rhythmic coordination training. 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW); 2022: IEEE.
67. Scalona E, De Marco D, Bazzini MC, Nuara A, Zilli A, Taglione E, et al. A repertoire of virtual-reality, occupational therapy exercises for motor rehabilitation based on action observation. Data. 2022;7(1):9.
68. Rebenitsch L, Owen C. Review on cybersickness in applications and visual displays. Virtual reality. 2016;20:101-25.