Multimodal Communication in Virtual and Face-to-Face Settings: Gesture Production and Speech Disfluency

Yıl 2024, Cilt: 44 Sayı: 3, 349 - 363, 16.12.2024

Burcu Arslan , Can Avcı , Demet Özer

https://doi.org/10.26650/SP2024-1443983

Öz

The COVID-19 pandemic has made online data collection a popular choice. It is important to evaluate how comparable online studies are to face-to-face studies, particularly in multimodal language research where modes of communication significantly impact the results. In this study, we examined individuals’ rates and patterns of speech disfluency and gesture use across face-to-face and online videoconferencing settings as they described their daily routines (N = 64). We asked whether and how multimodal language is affected across different communication settings and gesture use, particularly iconic gestures, is associated with speech fluency regardless of the context. Our results have showed that the participants’ overall disfluency rate was higher for the speech communicated via videoconferencing than the speech communicated face-to-face. However, the type of disfluencies changed across contexts, such that filled pauses and repairs were more common in online communication, whereas silent pauses were more common in face-to-face communication. These f indings signal an interplay between the cognitive functions of different disfluency types and communicative strategies. Results indicate that the overall gesture frequency and iconic gesture use were similar in both settings. Furthermore, the use of iconic gestures was found to negatively predict the overall disfluency rate, regardless of the setting. This finding suggests that using iconic gestures might facilitate cognitive processes, paving the way for a more fluent speech. This study demonstrates that multimodal language and communication strategies may vary across different communication settings and nuanced understanding of the differences in multimodal language between online and face-to-face communication can be gained using different contexts. The findings contribute to understanding the impact of increasingly widespread online communication on multimodal language production processes and provide foundation for future research.

Anahtar Kelimeler

Gesture production, speech disfluency, virtual communication, face-to-face communication

Çevrimiçi ve Yüz Yüze İletişim Ortamlarında Multimodal Dil Kullanımı: Jest Üretimi ve Konuşma Akıcılığı Üzerine Bir Araştırma

Öz

Çevrimiçi veri toplama, COVID-19 salgını nedeniyle öne çıkan bir seçenek haline gelmiştir. Çevrimiçi çalışmaların, özellikle bağlamın çok önemli bir etkiye sahip olduğu dil ve iletişim alanlarında, yüz yüze yapılan çalışmalarla ne ölçüde karşılaştırılabileceğini anlamak çok önemlidir. Bu çalışma, yüz yüze ve video konferans ortamlarında multimodal iletişimi araştırmak amacıyla, kişilerin (N= 64) günlük rutinlerini anlatırken kullandıkları konuşma akıcılıklarına ve sözlü dile eşlik eden jest üretimlerine odaklanmaktadır. Çalışmada, el jestlerinin ve sözlü dildeki akıcılığın farklı iletişim ortamlarında (çevrim içi ve yüz yüze) nasıl değiştiği ve iletişim ortamından bağımsız olarak jest kullanımının (özellikle ikonik jestlerin) konuşma akıcılığıyla ilişkisi araştırılmaktadır. Çalışmanın sonuçları, konuşma akışındaki bozulma oranının video konferans yoluyla iletişim kuranlarda, yüz yüze iletişim kuranlara göre daha yüksek olduğunu göstermiştir. Fakat konuşma akıcılığındaki farklı bozulma türlerinin iki ortamda farklılık gösterdiği bulunmuştur. Konuşmacıların video konferans ortamında daha fazla dolgulu duraksama ve onarım kullanırken, yüz yüze iletişim ortamında daha fazla sessiz duraksama kullandığı bulunmuştur. Bu bulgular, konuşmanın akıcılığındaki bozulmaların iletişim ortamına göre değişebileceğini ve farklı iletişim stratejileri doğurabileceğini göstermektedir. Bunun yanında, genel jest kullanımının ve özel olarak ikonik (temsili) jest kullanımının iki ortam arasında fark göstermediği bulunmuştur. Ayrıca, iletişim ortamından bağımsız olarak, ikonik jest kullanım sıklığının konuşma akıcılığını artırdığı bulunmuştur. Bu bulgu, özellikle ikonik jest kullanımının bilişsel süreçleri kolaylaştırarak daha akıcı bir konuşmaya zemin hazırlayabileceğini göstermektedir. Bu çalışma, multimodal dil ve iletişim stratejilerinin farklı iletişim ortamlarında değişebildiğini göstermekte ve bu anlamda bağlamın araştırılmasının önemini vurgulamaktadır. Sonuçlar, günümüzde özellikle yaygınlaşan çevrimiçi iletişimin multimodal dil üretim süreçleri üzerindeki etkilerini anlamaya katkı sağlamaktadır ve ileride yapılacak çalışmalar için önemli bir temel sunmaktadır.

Anahtar Kelimeler

Jest kullanımı, konuşmanın akıcılığının bozulması, çevrimiçi iletişim, yüz yüze iletişim

Kaynakça

• Alibali, M. W. (2005). Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. Spatial Cognition and Computation, 5(4), 307-331. https://doi.org/10.1207/ s15427633scc0504_2 google scholar
• Archibald, M. M., Ambagtsheer, R. C., Casey, M. G., & Lawless, M. (2019). Using zoom videoconferencing for qualitative data collection: Perceptions and experiences of researchers and participants. International Journal of Qualitative Methods, 18, 1-8. https://doi.org/10.1177/ 1609406919874596 google scholar
• Arslan, B., & Göksun, T. (2021). Ageing, working memory, and mental imagery: Understanding gestural communication in younger and older adults. Quarterly Journal of Experimental Psychology, 74(1), 29-44. https://doi.org/10.1177/1747021820944696 google scholar
• Arslan, B., & Göksun, T. (2022). Aging, gesture production, and disfluency in speech: A comparison of younger and older adults. Cognitive Science, 46(2), Article e13098. https://doi.org/10.1111/ cogs.13098 google scholar
• Arslan, B., Avcı, C., Yılmaztekin, A., & Göksun, T. (2024). Do bilingual adults gesture when they are disfluent?: Understanding gesture-speech interaction across first and second languages. Language, Cognition and Neuroscience, 39(5), 571-583. https://doi.org/10.1080/23273798. 2024.2345306 google scholar
• Avcı, C., Arslan, B., & Göksun, T. (2022). Gesture and speech disfluency in narrative context: Disfluency rates in spontaneous, restricted, and encouraged gesture conditions. In Culbertson J., Perfors A., Rabagliati H., & Ramenzoni V. (Eds.), Proceedings of the 44th annual conference of the cognitive science society (pp. 1912-1917). Cognitive Science Society. google scholar
• Bailenson, J. N. (2021). Nonverbal overload: A theoretical argument for the causes of Zoom fatigue. Technology, Mind, and Behavior, 2(1), 1-6. https://doi.org/10.1037/tmb0000030 google scholar
• Bock, K. (1996). Language production: Methods and methodologies. Psychonomic Bulletin & Review, 3, 395-421. https://doi.org/10.3758/BF03214545 google scholar
• Bortfeld, H., Leon, S. D., Bloom, J. E., Schober, M. F., & Brennan, S. E. (2001). Disfluency rates in conversation: Effects of age, relationship, topic, role, and gender. Language and Speech, 44(2), 123-147. https://doi.org/10.1177/00238309010440020101 google scholar
• Corley, M., & Stewart, O. W. (2008). Hesitation disfluencies in spontaneous speech: The meaning of um. Language and Linguistics Compass, 2(4), 589-602. https://doi.org/10.1111/j.1749-818X. 2008.00068.x google scholar
• Cserzo, D. (2021). Discourses andpractices of attention in video chat. Multimodal Communication, 10(2), 143-156. https://doi.org/10.1515/mc-2020-0010 google scholar
• Fraundorf, S. H., & Watson, D. G. (2014). Alice’s adventures in um-derland: Psycholinguistic sources of variation in disfluency production. Language, Cognition and Neuroscience, 29(9), 1083-1096. https://doi.org/10.1080/01690965.2013.832785 google scholar
• Geller, V., & Shaver, P. (1976). Cognitive consequences of self-awareness. Journal of Experimental Social Psychology, 12(1), 99-108. https://doi.org/10.1016/0022-1031(76)90089-5 google scholar
• Glassmeyer, D. M., & Dibbs, R. A. (2012). Researching from a distance: Using live web conferencing to mediate data collection. International Journal of Qualitative Methods, 11(3), 292-302. https: //doi.org/10.1177/160940691201100308 google scholar
• Graziano, M., & Gullberg, M. (2018). When speech stops, gesture stops: Evidence from developmental and crosslinguistic comparisons. Frontiers in Psychology, 9, Article e00879. https://doi.org/10.3389/fpsyg.2018.00879 google scholar
• Hassell, M. D., & Cotton, J. L. (2017). Some things are better left unseen: Toward more effective communication and team performance in video-mediated interactions. Computers in Human Behavior, 73, 200-208. https://doi.org/10.1016/j.chb.2017.03.039 google scholar
• Hyusein G., & Göksun, T. (2023). The creative interplay between hand gestures, convergent thinking, and mental imagery. PLOS ONE, 18(4), Article e0283859. https://doi.org/10.1371/journal.pone. 0283859 google scholar
• Kandemir, S., Özer, D., & Aktan-Erciyes, A. (2023). Multimodal language in child-directed versus adult-directed speech. Quarterly Journal of Experimental Psychology, 77(4), 716-728. https: //doi.org/10.1177/17470218231188832 google scholar
• Kısa, Y. D., Goldin-Meadow, S., & Casasanto, D. (2022). Do gestures really facilitate speech production?. Journal of Experimental Psychology: General, 151(6), 1252-1271. https://doi.org/ 10.1037/xge0001135 google scholar
• Kita, S., Alibali, M. W., & Chu, M. (2017). How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review, 124(3), 245-266. https: //doi.org/10.1037/rev0000059 google scholar
• Kita, S., & Davies, T. S. (2009). Competing conceptual representations trigger co-speech representational gestures. Language and Cognitive Processes, 24(5), 761-775. https://doi.org/ 10.1080/01690960802327971 google scholar
• Kita, S., & Özyürek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48(1), 16-32. https://doi.org/10.1016/ S0749-596X(02)00505-3 google scholar
• Krauss, R., Chen, Y., & Gottesman, R. (2000). Lexical gestures and lexical access: A process model. In D. McNeill (Ed.), Language and gesture: Window into thought and action (pp. 261-283). Cambridge University Press. google scholar
• Lausberg, H., & Sloetjes, H. (2009). Coding gestural behavior with the NEUROGES-ELAN system. Behavior Research Methods, 41(3), 841-849. https://doi.org/10.3758/BRM.41.3.841 google scholar
• Liebling, B. A., & Shaver, P. (1973). Evaluation, self-awareness, and task performance. Journal of Experimental Social Psychology, 9(4), 297-306. https://doi.org/10.1016/0022- 1031(73)90067- X google scholar
• Maclay, H., & Osgood, C. E. (1959). Hesitation phenomena in spontaneous English speech. Word, 15(1), 19-44. https://doi.org/10.1080/00437956.1959.11659682 google scholar
• McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago Press. google scholar
• Melinger, A., & Kita, S. (2007). Conceptualisation load triggers gesture production. Language and Cognitive Processes, 22(4), 473-500. https://doi.org/10.1080/01690960600696916 google scholar
• Morsella, E., & Krauss, R. M. (2004). The role of gestures in spatial working memory and speech. The American Journal of Psychology, 411-424. https://doi.org/10.2307/4149008 google scholar
• Oviatt, S. (1995). Predicting spoken disfluencies during human-computer interaction. Computer Speech and Language, 9(1), 19-36. doi:10.1006/csla.1995.0002 google scholar
• Özder, L. E., Özer D., & Göksun, T. (2022). Gesture use in L1-Turkish and L2-English: Evidence from emotional narrative retellings. Quarterly Journal of Experimental Psychology, 76(8), 1797-1816. https://doi.org/10.1177/17470218221126685 google scholar
• Özer, D., Tansan, M., Özer, E. E., Malykhina, K., Chatterjee, A., & Göksun, T. (2017). The effects of gesture restriction on spatial language in young and elderly adults. In G. Gunzelmann, A. Howes, T. Tenbrink, & E. Davelaar (Eds.), Proceedings of the 38th Annual Conference of the Cognitive Science Society (pp. 1471-1476). Cognitive Science Society. google scholar
• Rauscher, F. H., Krauss, R. M., & Chen, Y. (1996). Gesture, speech, and lexical access: The role of lexical movements in speech production. Psychological Science, 7(4), 226-231. https://doi.org/ 10.1111/j.1467-9280.1996.tb00364.x google scholar
• Seyfeddinipur, M. & Kita, S. (2001, August 29-31). Gesture as an indicator of early error detection in self-monitoring of speech. ISCA Tutorial and Research Workshop (ITRW) on Disfluency in Spontaneous Speech, Edinburgh, Scotland, UK. google scholar
• Silverman, S. W., & Ratner, N. B. (1997). Syntactic complexity, fluency, and accuracy of sentence imitation in adolescents. Journal of Speech, Language, and Hearing Research, 40(1), 95-106. https://doi.org/10.1044/jslhr.4001.95 google scholar
• Smith, V. L., & Clark, H. H. (1993). On the course of answering questions. Journal of Memory and Language, 32(1), 25-38. https://doi.org/10.1006/jmla.1993.1002 google scholar
• Ter Bekke, M., Drijvers, L., & Holler, J. (2024). Hand gestures have predictive potential during conversation: An investigation of the timing of gestures in relation to speech. Cognitive Science, 48(1), Article e13407. https://doi.org/10.1111/cogs.13407 google scholar
• Torrentira, M. C., Jr. (2020). Online data collection as adaption in conducting quantitative and qualitative research during the COVID-19 pandemic. European Journal of Education Studies, 7(11), 78-87. http://dx.doi.org/10.46827/ejes.v7i11.3336 google scholar
• Ünal, E., Manhardt, F., & Özyürek, A. (2022). Speaking and gesturing guide event perception during message conceptualization: Evidence from eye movements. Cognition, 225, Article e105127. https://doi.org/10.1016/j.cognition.2022.105127 google scholar
• Xu, Q., & Behring, D. (2014). The richer, the Better? Effects of modality on intercultural virtual collaboration. International Journal of Communication, 8, 2733-2754. google scholar
• Zoom Video Communications Inc. (2016). Security guide. Zoom Video Communications Inc. Retrieved from https://d24cgw3uvb9a9h.cloudfront.net/static/81625/doc/ Zoom- Security-White-Paper.pdf google scholar