Back to listing

Building better sign language technologies

Outreach, Technology

Date: 19 October 2021

TAP Project Manager Ben Foley convened ‘Voyages in Language Technologies’ again this year to students at the University of Queensland. One of the highlights of the 2021 course was a guest lecture by CoEDL Affiliate Jessica Korte on developing sign language technology to facilitate communication for Deaf people. With CoEDL support, Jess was able to present the lecture with the help of sign language interpreters to an audience of computer science students and members of the Deaf community.

Jess is leading the Auslan Communication Technologies Pipeline project. Supported by TASDCRC (Trusted Autonomous Systems Defence Cooperative Research Centre) — where Jess is Research Fellow — the three-year, three-part project will create a modular artificial intelligence system capable of recognising and processing conversational Auslan (Australian Sign Language) inputs and generating artificial but human-like Auslan outputs. 

Guiding Jess’ approach to the project is a passion for designing better technology that can improve lives, particularly for minority groups. This passion informs her design philosophy, which is centred on involving end users in the design process.

“The best way to make a new technology is to involve the end users in the design process,” Jess says. “That way we can ensure we are meeting real needs and supporting the real abilities of those end users.”

This approach was evident in her lecture, which began with an introduction to sign language and the Deaf community. Sign language is diverse — in Australia alone, there are two dialects of Auslan and countless Indigenous sign languages — as well as complex and multi-modal. Jess explains that a single sign can be deconstructed into five components: hand shape, orientation, location, movement and facial expression. She argues that technology and design must capture all these elements, while also understanding the conventions and needs of the community. 

Jess Korte’s lecture slides with sign language interpreter Adele (image: CoEDL/lecture screenshot).

Design processes that overlook parts of this information can result in what are sometimes called ‘Disability Dongles’. This term — coined by researcher and advocate Elize Jackson — refers to technology developed with the intention of solving a perceived challenge that nevertheless fails to meet or answer the needs of the target community. For example, by focusing on the hand as the mode of sign language, sign language gloves fail to capture the importance of facial expression. They also misconstrue the challenge sign language users face. Rather than meaningfully improving communication, accessibility or equity, these gloves can actually be burdensome for sign language users. 

Disability Dongles highlight the need to work closely with communities to understand how technology can best serve their needs. For this reason, Jess has assembled a Project Advisory Panel comprised of community members and advertised positions in her team on community forums to encourage Deaf applicants. 

“An important principle of this project is language-centredness and community-centredness: centring the realities of Auslan and Deaf people’s lived experiences.”

The project itself seeks to develop an ‘Alexa for Auslan’ — a personal assistive device Deaf and hard of hearing users can interact with through sign language. Three modules will make this possible. The Recognition module receives Auslan input from a sign language user. The Processing module decodes what is being signed and figures out the appropriate response. Then, the Production module visualises this response by generating an animated Auslan output.

It is a computationally complex task and Jess emphasises that her team is only in the prototype phase. But already there is a lot of enthusiasm for the project. One Project Advisory Panel member expressed his excitement during the question-and-answer session at the end of Jess’ Voyages in Technology lecture.

“I’m really excited to be involved in the project as a Deaf Panel Member and I can see that there’s so much possibility for the future,” he voiced through an interpreter. “About 20 years ago the technology was really awful for us, but now in 2021 there’s a lot of feeling that it’s possible and that we’re going to really thrive in this space…. What I see from the project is that you’re really championing Deaf people, and I wanted to say thank you for making us the champions of this project.”

A recording of Jess’ lecture — Voyages in Language Technology: Sign Languages and Communicative Gesture — is available here.



Header — Jess Korte’s lecture slides, featuring a quote from CoEDL Affiliate Gabrielle Hodge, with sign language interpreter Alisa (image: CoEDL/lecture screenshot).

Image 2 — Jess Korte’s lecture slides with sign language interpreter Adele (image: CoEDL/lecture screenshot). 

  • Australian Government
  • The University of Queensland
  • Australian National University
  • The University of Melbourne
  • Western Sydney University