Feedback signals are ubiquitous in interactions, one might expect them to vary considerably across individuals, across different types of contexts (spontaneous vs. task-oriented conversations, the amount of social distance between the partners) and more importantly across language modalities (signed vs. spoken). The perceptual system of sign languages relies heavily on visual input and we might also expect modality specific differences in the use of feedback signals.

We study the mechanisms recruited for feedback in two language modalities and address the compositional nature of such responses in two spoken and two signed languages in natural dyadic interaction. We use the same methodology and the same analysis of signed and spoken language data and analyze the type of given feedback. We differentiate between vocal, manual, nonmanual feedback and a combination of these.

One of the most important features of the research is the cross-linguistics and cross-modal approach which is a critical factor to get a better understanding of human interaction. We argue that the analysis of language feedback clearly requires a multimodal approach.

Our first study has shown that feedback signals in four languages (German Sign Language, Russian Sign Language, German and Russian), include a nonmanual component. Unimodal feedback signals (i.e. consisting of only a manual or a vocal element) are extremely rare across languages and modalities (light grey and purple boxes). In the two sign languages (DGS, RSL), feedback signals are most frequently produced only nonmanually (green boxes). In the two spoken languages (GER, RUS), the most frequent type of feedback signals contains a verbal and a nonmanual component (brown boxes).