Çevrimiçi ve Yüz Yüze İletişim Ortamlarında Multimodal Dil Kullanımı: Jest Üretimi ve Konuşma Akıcılığı Üzerine Bir Araştırma

Abstract

Çevrimiçi veri toplama, COVID-19 salgını nedeniyle öne çıkan bir seçenek haline gelmiştir. Çevrimiçi çalışmaların, özellikle bağlamın çok önemli bir etkiye sahip olduğu dil ve iletişim alanlarında, yüz yüze yapılan çalışmalarla ne ölçüde karşılaştırılabileceğini anlamak çok önemlidir. Bu çalışma, yüz yüze ve video konferans ortamlarında multimodal iletişimi araştırmak amacıyla, kişilerin (N= 64) günlük rutinlerini anlatırken kullandıkları konuşma akıcılıklarına ve sözlü dile eşlik eden jest üretimlerine odaklanmaktadır. Çalışmada, el jestlerinin ve sözlü dildeki akıcılığın farklı iletişim ortamlarında (çevrim içi ve yüz yüze) nasıl değiştiği ve iletişim ortamından bağımsız olarak jest kullanımının (özellikle ikonik jestlerin) konuşma akıcılığıyla ilişkisi araştırılmaktadır. Çalışmanın sonuçları, konuşma akışındaki bozulma oranının video konferans yoluyla iletişim kuranlarda, yüz yüze iletişim kuranlara göre daha yüksek olduğunu göstermiştir. Fakat konuşma akıcılığındaki farklı bozulma türlerinin iki ortamda farklılık gösterdiği bulunmuştur. Konuşmacıların video konferans ortamında daha fazla dolgulu duraksama ve onarım kullanırken, yüz yüze iletişim ortamında daha fazla sessiz duraksama kullandığı bulunmuştur. Bu bulgular, konuşmanın akıcılığındaki bozulmaların iletişim ortamına göre değişebileceğini ve farklı iletişim stratejileri doğurabileceğini göstermektedir. Bunun yanında, genel jest kullanımının ve özel olarak ikonik (temsili) jest kullanımının iki ortam arasında fark göstermediği bulunmuştur. Ayrıca, iletişim ortamından bağımsız olarak, ikonik jest kullanım sıklığının konuşma akıcılığını artırdığı bulunmuştur. Bu bulgu, özellikle ikonik jest kullanımının bilişsel süreçleri kolaylaştırarak daha akıcı bir konuşmaya zemin hazırlayabileceğini göstermektedir. Bu çalışma, multimodal dil ve iletişim stratejilerinin farklı iletişim ortamlarında değişebildiğini göstermekte ve bu anlamda bağlamın araştırılmasının önemini vurgulamaktadır. Sonuçlar, günümüzde özellikle yaygınlaşan çevrimiçi iletişimin multimodal dil üretim süreçleri üzerindeki etkilerini anlamaya katkı sağlamaktadır ve ileride yapılacak çalışmalar için önemli bir temel sunmaktadır.

Keywords

• Jest kullanımı
• konuşmanın akıcılığının bozulması
• çevrimiçi iletişim
• yüz yüze iletişim

Multimodal Communication in Virtual and Face-to-Face Settings: Gesture Production and Speech Disfluency

Abstract

The COVID-19 pandemic has made online data collection a popular choice. It is important to evaluate how comparable online studies are to face-to-face studies, particularly in multimodal language research where modes of communication significantly impact the results. In this study, we examined individuals’ rates and patterns of speech disfluency and gesture use across face-to-face and online videoconferencing settings as they described their daily routines (N = 64). We asked whether and how multimodal language is affected across different communication settings and gesture use, particularly iconic gestures, is associated with speech fluency regardless of the context. Our results have showed that the participants’ overall disfluency rate was higher for the speech communicated via videoconferencing than the speech communicated face-to-face. However, the type of disfluencies changed across contexts, such that filled pauses and repairs were more common in online communication, whereas silent pauses were more common in face-to-face communication. These f indings signal an interplay between the cognitive functions of different disfluency types and communicative strategies. Results indicate that the overall gesture frequency and iconic gesture use were similar in both settings. Furthermore, the use of iconic gestures was found to negatively predict the overall disfluency rate, regardless of the setting. This finding suggests that using iconic gestures might facilitate cognitive processes, paving the way for a more fluent speech. This study demonstrates that multimodal language and communication strategies may vary across different communication settings and nuanced understanding of the differences in multimodal language between online and face-to-face communication can be gained using different contexts. The findings contribute to understanding the impact of increasingly widespread online communication on multimodal language production processes and provide foundation for future research.

Keywords

• Gesture production
• speech disfluency
• virtual communication
• face-to-face communication

References

1. Alibali, M. W. (2005). Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. Spatial Cognition and Computation, 5(4), 307-331. https://doi.org/10.1207/ s15427633scc0504_2 google scholar
2. Archibald, M. M., Ambagtsheer, R. C., Casey, M. G., & Lawless, M. (2019). Using zoom videoconferencing for qualitative data collection: Perceptions and experiences of researchers and participants. International Journal of Qualitative Methods, 18, 1-8. https://doi.org/10.1177/ 1609406919874596 google scholar
3. Arslan, B., & Göksun, T. (2021). Ageing, working memory, and mental imagery: Understanding gestural communication in younger and older adults. Quarterly Journal of Experimental Psychology, 74(1), 29-44. https://doi.org/10.1177/1747021820944696 google scholar
4. Arslan, B., & Göksun, T. (2022). Aging, gesture production, and disfluency in speech: A comparison of younger and older adults. Cognitive Science, 46(2), Article e13098. https://doi.org/10.1111/ cogs.13098 google scholar
5. Arslan, B., Avcı, C., Yılmaztekin, A., & Göksun, T. (2024). Do bilingual adults gesture when they are disfluent?: Understanding gesture-speech interaction across first and second languages. Language, Cognition and Neuroscience, 39(5), 571-583. https://doi.org/10.1080/23273798. 2024.2345306 google scholar
6. Avcı, C., Arslan, B., & Göksun, T. (2022). Gesture and speech disfluency in narrative context: Disfluency rates in spontaneous, restricted, and encouraged gesture conditions. In Culbertson J., Perfors A., Rabagliati H., & Ramenzoni V. (Eds.), Proceedings of the 44th annual conference of the cognitive science society (pp. 1912-1917). Cognitive Science Society. google scholar
7. Bailenson, J. N. (2021). Nonverbal overload: A theoretical argument for the causes of Zoom fatigue. Technology, Mind, and Behavior, 2(1), 1-6. https://doi.org/10.1037/tmb0000030 google scholar
8. Bock, K. (1996). Language production: Methods and methodologies. Psychonomic Bulletin & Review, 3, 395-421. https://doi.org/10.3758/BF03214545 google scholar

9. Bortfeld, H., Leon, S. D., Bloom, J. E., Schober, M. F., & Brennan, S. E. (2001). Disfluency rates in conversation: Effects of age, relationship, topic, role, and gender. Language and Speech, 44(2), 123-147. https://doi.org/10.1177/00238309010440020101 google scholar
10. Corley, M., & Stewart, O. W. (2008). Hesitation disfluencies in spontaneous speech: The meaning of um. Language and Linguistics Compass, 2(4), 589-602. https://doi.org/10.1111/j.1749-818X. 2008.00068.x google scholar
11. Cserzo, D. (2021). Discourses andpractices of attention in video chat. Multimodal Communication, 10(2), 143-156. https://doi.org/10.1515/mc-2020-0010 google scholar
12. Fraundorf, S. H., & Watson, D. G. (2014). Alice’s adventures in um-derland: Psycholinguistic sources of variation in disfluency production. Language, Cognition and Neuroscience, 29(9), 1083-1096. https://doi.org/10.1080/01690965.2013.832785 google scholar
13. Geller, V., & Shaver, P. (1976). Cognitive consequences of self-awareness. Journal of Experimental Social Psychology, 12(1), 99-108. https://doi.org/10.1016/0022-1031(76)90089-5 google scholar
14. Glassmeyer, D. M., & Dibbs, R. A. (2012). Researching from a distance: Using live web conferencing to mediate data collection. International Journal of Qualitative Methods, 11(3), 292-302. https: //doi.org/10.1177/160940691201100308 google scholar
15. Graziano, M., & Gullberg, M. (2018). When speech stops, gesture stops: Evidence from developmental and crosslinguistic comparisons. Frontiers in Psychology, 9, Article e00879. https://doi.org/10.3389/fpsyg.2018.00879 google scholar
16. Hassell, M. D., & Cotton, J. L. (2017). Some things are better left unseen: Toward more effective communication and team performance in video-mediated interactions. Computers in Human Behavior, 73, 200-208. https://doi.org/10.1016/j.chb.2017.03.039 google scholar
17. Hyusein G., & Göksun, T. (2023). The creative interplay between hand gestures, convergent thinking, and mental imagery. PLOS ONE, 18(4), Article e0283859. https://doi.org/10.1371/journal.pone. 0283859 google scholar
18. Kandemir, S., Özer, D., & Aktan-Erciyes, A. (2023). Multimodal language in child-directed versus adult-directed speech. Quarterly Journal of Experimental Psychology, 77(4), 716-728. https: //doi.org/10.1177/17470218231188832 google scholar
19. Kısa, Y. D., Goldin-Meadow, S., & Casasanto, D. (2022). Do gestures really facilitate speech production?. Journal of Experimental Psychology: General, 151(6), 1252-1271. https://doi.org/ 10.1037/xge0001135 google scholar
20. Kita, S., Alibali, M. W., & Chu, M. (2017). How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review, 124(3), 245-266. https: //doi.org/10.1037/rev0000059 google scholar
21. Kita, S., & Davies, T. S. (2009). Competing conceptual representations trigger co-speech representational gestures. Language and Cognitive Processes, 24(5), 761-775. https://doi.org/ 10.1080/01690960802327971 google scholar
22. Kita, S., & Özyürek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48(1), 16-32. https://doi.org/10.1016/ S0749-596X(02)00505-3 google scholar
23. Krauss, R., Chen, Y., & Gottesman, R. (2000). Lexical gestures and lexical access: A process model. In D. McNeill (Ed.), Language and gesture: Window into thought and action (pp. 261-283). Cambridge University Press. google scholar
24. Lausberg, H., & Sloetjes, H. (2009). Coding gestural behavior with the NEUROGES-ELAN system. Behavior Research Methods, 41(3), 841-849. https://doi.org/10.3758/BRM.41.3.841 google scholar
25. Liebling, B. A., & Shaver, P. (1973). Evaluation, self-awareness, and task performance. Journal of Experimental Social Psychology, 9(4), 297-306. https://doi.org/10.1016/0022- 1031(73)90067- X google scholar
26. Maclay, H., & Osgood, C. E. (1959). Hesitation phenomena in spontaneous English speech. Word, 15(1), 19-44. https://doi.org/10.1080/00437956.1959.11659682 google scholar
27. McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago Press. google scholar
28. Melinger, A., & Kita, S. (2007). Conceptualisation load triggers gesture production. Language and Cognitive Processes, 22(4), 473-500. https://doi.org/10.1080/01690960600696916 google scholar
29. Morsella, E., & Krauss, R. M. (2004). The role of gestures in spatial working memory and speech. The American Journal of Psychology, 411-424. https://doi.org/10.2307/4149008 google scholar
30. Oviatt, S. (1995). Predicting spoken disfluencies during human-computer interaction. Computer Speech and Language, 9(1), 19-36. doi:10.1006/csla.1995.0002 google scholar
31. Özder, L. E., Özer D., & Göksun, T. (2022). Gesture use in L1-Turkish and L2-English: Evidence from emotional narrative retellings. Quarterly Journal of Experimental Psychology, 76(8), 1797-1816. https://doi.org/10.1177/17470218221126685 google scholar
32. Özer, D., Tansan, M., Özer, E. E., Malykhina, K., Chatterjee, A., & Göksun, T. (2017). The effects of gesture restriction on spatial language in young and elderly adults. In G. Gunzelmann, A. Howes, T. Tenbrink, & E. Davelaar (Eds.), Proceedings of the 38th Annual Conference of the Cognitive Science Society (pp. 1471-1476). Cognitive Science Society. google scholar
33. Rauscher, F. H., Krauss, R. M., & Chen, Y. (1996). Gesture, speech, and lexical access: The role of lexical movements in speech production. Psychological Science, 7(4), 226-231. https://doi.org/ 10.1111/j.1467-9280.1996.tb00364.x google scholar
34. Seyfeddinipur, M. & Kita, S. (2001, August 29-31). Gesture as an indicator of early error detection in self-monitoring of speech. ISCA Tutorial and Research Workshop (ITRW) on Disfluency in Spontaneous Speech, Edinburgh, Scotland, UK. google scholar
35. Silverman, S. W., & Ratner, N. B. (1997). Syntactic complexity, fluency, and accuracy of sentence imitation in adolescents. Journal of Speech, Language, and Hearing Research, 40(1), 95-106. https://doi.org/10.1044/jslhr.4001.95 google scholar
36. Smith, V. L., & Clark, H. H. (1993). On the course of answering questions. Journal of Memory and Language, 32(1), 25-38. https://doi.org/10.1006/jmla.1993.1002 google scholar
37. Ter Bekke, M., Drijvers, L., & Holler, J. (2024). Hand gestures have predictive potential during conversation: An investigation of the timing of gestures in relation to speech. Cognitive Science, 48(1), Article e13407. https://doi.org/10.1111/cogs.13407 google scholar
38. Torrentira, M. C., Jr. (2020). Online data collection as adaption in conducting quantitative and qualitative research during the COVID-19 pandemic. European Journal of Education Studies, 7(11), 78-87. http://dx.doi.org/10.46827/ejes.v7i11.3336 google scholar
39. Ünal, E., Manhardt, F., & Özyürek, A. (2022). Speaking and gesturing guide event perception during message conceptualization: Evidence from eye movements. Cognition, 225, Article e105127. https://doi.org/10.1016/j.cognition.2022.105127 google scholar
40. Xu, Q., & Behring, D. (2014). The richer, the Better? Effects of modality on intercultural virtual collaboration. International Journal of Communication, 8, 2733-2754. google scholar
41. Zoom Video Communications Inc. (2016). Security guide. Zoom Video Communications Inc. Retrieved from https://d24cgw3uvb9a9h.cloudfront.net/static/81625/doc/ Zoom- Security-White-Paper.pdf google scholar