MULTIMODAL PERCEPTION OF PRAISING AND BLAMING MANDARIN SPEECH BETWEEN THE INTERLOCUTORS WITH FRIENDLY OR HOSTILE RELATIONSHIPS Shanpeng Li 1 and Wentao Gu 1,2 1 School of Chinese Language and Literature, Nanjing Normal University 2 Nanjing Normal University and The George Washington University Joint Laboratory of Speech, Hearing and Rehabilitation Sciences [email protected], [email protected]ABSTRACT This work investigated the role of multimodality and relationship between interlocutors in perception of Mandarin praising and blaming attitudes. The perceptual experiment found that the differences of accuracy between praising and blaming attitude were larger in friendly relationship than that in hostile. Moreover, praising attitude was identified better in audio channel than that in visual channel, while blaming attitude was discriminated more in visual channel than that in audio channel. This suggests that the hostile relationship between interlocutors was conveyed mainly by facial expression, either in praising or blaming speech. Keywords: multimodal speech perception, speaker relationship, attitude, praising, blaming. 1. INTRODUCTION Although there have been a number of studies on the processing of emotions, little has been known about how the audio and visual channels contribute to the perception of social affects such as praising and blaming. Moreover, while most studies of emotions and social affects are focused on English, French, or Japanese, there are fewer studies on Mandarin. Therefore, this work is aimed to the perception of Mandarin attitudinal speech. The perception of attitudinal speech is affected by many factors, such as sentence length, modality, and linguistic background of the listener. Lu et al. [6] found that sentence length could affect the identification rate of attitudes in Mandarin: the shorter sentence leads to the lower perceptual identification rate, except for the infant-directed speech. The effect of sentence length was also found in Japanese [10], though less regular. Gu et al. [4] found that the valence of attitudes also played roles in the perception of Mandarin attitudes. They reported that the positive attitude had more accuracy than the negative attitude only in the praising- blaming pairs. On the contrast, other attitude pairs, such as friendly-hostile, polite-rude, serious-joking, and confident-uncertain had a higher accuracy in negative than positive. Apart from these linguistic factors referred in previous studies, non-linguistic cues, like the role of relationship of interlocutor, has received less attention, where we just want to focus on. Many studies of attitudes have been conducted (cf. [1, 2, 8]), including some cross-linguistic studies (cf. [11]), but most studies only dealt with the audio modality. Different modalities of stimuli can also change the result of perception. Specifically, audio and visual signals can transmit speech and facial expression about affects. When listeners identify attitudes, they can use different modalities to make a decision, like tone of voice, facial expression, and body gesture. de Moraes et al. [3] investigated the role of multimodality in the perception of Brazilian Portuguese attitudes. Their results showed that for propositional attitudes, audio and visual modalities were equivalent, except for irony, while for social attitudes, audio played a less important role, receiving significantly lower scores than audio- visual and visual for all attitudes except seduction and politeness. Also, Hönemann et al. [5] found that, in German, audio played more important roles than audio-visual modality in declarative sentence and irritation. The aim of this work is to study whether the relationships between interlocutors can influence the multimodal perception of praising and blaming attitudes in Mandarin speech. In the present study, we only examine one relationship between interlocutors, i.e., friendly and hostile. When speaker and listener are friendly, we called praising attitude as friendly praising while the blaming speech as friendly blaming. When speaker and listener are hostile, we called praising attitude as hostile praising while blame speech as hostile blaming. So totally, we discuss these four speech styles (2 relationships * 2 attitudes) to investigate the role of speaker relationship in the multimodal perception of praising and blaming Mandarin speech. 3533
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
MULTIMODAL PERCEPTION OF PRAISING AND BLAMING MANDARIN
SPEECH BETWEEN THE INTERLOCUTORS WITH FRIENDLY OR
HOSTILE RELATIONSHIPS
Shanpeng Li1 and Wentao Gu1,2
1School of Chinese Language and Literature, Nanjing Normal University
2Nanjing Normal University and The George Washington University Joint Laboratory of Speech, Hearing and