Course abstract - Kelly & Green

Course: Multimodal communication: tools and techniques for analysis

Instructors: Barb Kelly, Jenny Green (University of Melbourne)

Description: This course gives a brief introduction to multimodal communication and the tools and technologies used in the study of this broad field. The driving questions underpinning this course are ‘How have technologies changed the way we conduct research into multimodal communication, and, in turn, how are the results of this research reflected in new technologies?’ We explore the concept of the ‘uncanny valley’ in multimodal communication and view this from differing multidisciplinary perspectives.

We first survey the field of multimodal studies, examine how it has changed over time, and look at the influence of technological developments on the field. We touch on diverse case studies including those on co-speech gesture, on the role of communicative action in child language acquisition, and the ways that shared repertoires of action emerge in communities where sign languages and ‘gesture’ co-exist. The focus then turns to data collection practices, including ethical issues in multimodal research, filming techniques, and the use of tools for annotation and coding. The course will be structured around hands-on ELAN training using a dataset provided for in-class use (participants can also use their own data).

In light of students’ own experience of coding multimodal data sets in the class we then examine how academic research on multimodal communication informs different research fields and feeds into the development of new technologies. In particular we consider the ways that research on gesture and bodily action influences work on social robotics, gaming and avatar development.

Requirements for participants: No background is assumed. (1) Bring a laptop with the latest version of ELAN installed. (2) Read the ELAN coding manual 

Limited registrations: This course will be capped at 20 students. (Students will be selected randomly after registrations close from those who select this course as their first preference. Registrants who select this course will be asked to select a second course preference from the same time slot).

  • Australian Government
  • The University of Queensland
  • Australian National University
  • The University of Melbourne
  • Western Sydney University