Multimodal Patterns of Speech Disfluencies Compensators (Based on Student Dialogic Discourse)
DOI: 10.23951/1609-624X-2023-1-9-18
Speech as a multidimensional phenomenon still remains relevant in linguistics studies. It is known that verbal communication evolved later than non-verbal one, however, a multimodal approach to language communication modeling based on spontaneous dialogues is only gaining momentum in Russian linguistics studies. The aim of the article is to provide data on principles of verbal and non-verbal correlation at times of difficulty during the development of spoken discourse between two speakers, as well as to systematize them in the most frequent multimodal patterns. The material of the article is 16 video clips of spontaneous dialogues of Russian native speakers, the total timing is 1.5 hours. The annotated video materials are students’ spontaneous dialogic discourse. The topics of dialogues can be divided into two groups – communication between friends/colleagues and communication between the client and service staff. Each video clip is annotated in ELAN v.6.2. The presented results are based on general linguistic methods of description and comparison, as well as a method of analyzing a cognitive event. Analyzing cognitive events and providing statistics on the subject, we describe the main multimodal patterns, which have been classified by the frequent nature of some speech disfluencies and non-verbal response to them. Our study is based on the gestural repetition concept. As a result, the most common speech disfluencies have been identified and divided into five large blocks. Each of the blocks is accompanied by the main multimodal patterns, implemented in different channels – manual channel, gaze channel and cephalic channel. The number of speech disfluencies recorded at the external level directly depends on the topic and not always on the level of competence of the speakers, since all dialogues are improvised by students who only have presuppositions regarding the deployment of discourse depending on the given role. Speech disfluencies and the principles of their compensation in different multimodal channels depend on both internal (cognitive) and external factors (overlapping and interruptions), which represent an interpenetrating indivisible system.
Keywords: multimodal pattern, speech disfluency, manual channel, gaze channel, cephalic channel, cognitive event, overlapping
References:
1. Napier J. Hands. US.: Princeton University Press, 1993. 200 p.
2. Podlesskaya V. I., Kirbik A.A. Samoispravleniya govoryashchego i drugiye tipy rechevykh sboyev kak ob’’yekt annotirovaniya v korpusakh ustnoy rechi [Speaker’s self-corrections and other types of speech disfluencies as an object of annotation in spoken corpora]. Nauchno-tekhnicheskaya informatsiya. Seriya 2: Informatsionnyye protsessy i sistemy – Scientific and technical information. Series 2: Information Processes and Systems, 2007, no. 2, pp. 2–23 (in Russian).
3. Akhutina T. V. Porozhdeniye rechi. Neyrolingvisticheskiy analiz sintaksisa [Generation of speech. Neurolinguistic analysis of syntax]. Moscow, Yazyki Slavyanskoy kul’tury Publ., 1989. 424 p. (in Russian).
4. Dell G. A. Spreading-activation Theory of Retrieval in Sentence production. Psychological Review, 1986, vol. 93, pp. 283–321.
5. Krasikov Yu. V. Teoriya rechevykh oshibok (na materiale oshibok naborshchika) [Theory of speech errors (on the typesetter’s errors material)]. Moscow, Nauka Publ., 1980. 124 p. (in Russian).
6. Levelt W. J. M. Monitoring and Self-repair in Speech. Cognition. 1983. Vol. 14. Р. 41–104.
7. Leont’ev A. A. Psikholingvisticheskiye edinitsy i porozhdeniye rechevogo vyskazyvaniya [Psycholinguistic units and the generation of speech utterance]. Moscow, Nauka Publ., 2003. 250 p. (in Russian).
8. Postma A. Detection of Errors during Speech Production: A Review of Speech Monitoring Models. Cognition, 2000, vol. 77, pp. 97–131.
9. Eisler F. G. Psycholinguistics: Experiments in Spontaneous Speech. Academic Press Inc., 1968. 169 p.
10. Kendon A. Origins of Modern Gesture Studies. Gesture and the Dynamic Dimension of Language. Eds. S. Duncan, J. Cassell, Elena T. Levy. 2007. Pp. 1–11.
11. Calbris G. Elements of Meaning in Gesture. Amsterdam; Philadelphia (PA), 2011. 378 p.
12. Loehr D. P. Temporal, Structural and Pragmatic Synchrony between Intonation and Gesture. Laboratory Phonology, 2012, vol. 3 (1), pp. 71–89.
13. Graf H. P. et al. Visual prosody: Facial Movements Accompanying Speech. Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, 2002, pp. 396–401.
14. Fung H., Mok P. Temporal Coordination between Focus Prosody and Pointing Gestures in Cantonese. Journal of Phonetics, 2018, vol. 71, pp. 113–125.
15. Korotayev N. A. et al. Disfluencies in Russian Spoken Monologues: A Distributional Analysis. Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialogue 2020”, 2020. Pp. 454–466.
16. Watzlawick P., Beavin J. H., Jackson Don D. Pragmatics of Human Communication. A Study of Interactional Patterns, Pathologies and Paradoxes. London, Faber and Faber, 1968. 296 p.
17. Leonov I. V. Patterny kul’turno-istoricheskogo protsessa: paradigmal’no-tematicheskiy analiz. Avtoref. dis. dokt. kul’turologii [Patterns of the cultural-historical process: paradigm-thematic analysis. Abstract of thesis. doc. cultural studies]. Saint Petersburg, 2015. 41 p. (in Russian).
18. Chistova E. V. Interactivity of simultaneous interpreters as actors of a cognitive event. J. Sib. Fed. Univ. Humanit. Soc. Sci., 2020, vol. 13 (3), pp. 375–384.
19. Kolmogorova A. V., Chistova E. V. Kognitivnyy mekhanizm porozhdeniya kreativnykh resheniy v perevodcheskoy deyatel’nosti [Cognitive Basis of Making Creative Decisions in Translation Activities]. Vestnik Volgogradskogo gosudarstvennogo universiteta – Science Journal of Volgograd State University, 2020, no. 3, pp. 59–71 (in Russian).
20. Kolmogorova A. V. Languaging: yazyk v tochke peresecheniya kommunikatsii i kognitsii [Languaging: Language at the intersection of communication and cognition]. Kognitivnyye issledovaniya yazyka – Cognitive Research Studies, 2017, no. 29, pp. 95–103 (in Russian).
21. Adda G. et al. Speech Overlap and Interplay with Disfluencies in Political Interviews, 2007. Pp. 41–46.
( 2.49 MB ) 
( 2.46 MB )
Issue: 1, 2023
Series of issue: Issue 1
Rubric: THEORETICAL AND APPLIED LINGUISTICS
Pages: 9 — 18
Downloads: 428